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Introduction 



The need to improve the training of persons who can systematically 
work toward solutions cf major educational problems is widely recognized 
within the academic community. There is an increasing demand for person- 
nel to assess the status of students and their backgtounds, to design new 
and imaginative instructional programs and materials, to evaluate existing 
and newly introduced courses of study, to assess the quality of the 
education offer, and to interpret and plan the utilization of researcli 
findings . 

A variety of evidence indicated, however, that many persons per- 
forming in research and research- related roles are severely handicapped 
by their lack of skill or knov.*ledge in relation to substantive and 
methodological requisites relevant to their areas of investigation. 
Moreover, individuals initially ^v-ell- trained to engage in their 
respective research activities are rapidly faced with obsolescence, 
for this field appears almost devoid of viable programs designed to 
keep such persons abreast of new developments in their areas of special- 
ization. 

Therefore 5 the principal objective of this grant was to develop 
new ways in which scientific research associdtiohs can create coordinated 
continued, and more systematic approaches to research training. Research , 
for the purposes of this grant, was defined in its broadest sense to 
include both development and evaluation. The conceptual framework 
and organizational procedures by which a professional organization (AERA) 
performed this task not only serves as a prototype for other institutions, 
organizations, and agencies to develop creative modes of training, but 
generated new knowledge about training and retraining of research and 



research related personnel in education. 

The types of training activities initiated by aERA were designed 
to (1) prevent potenuial obsolescence among educational researchers 
with respect to developed metnodology and technology, (2) improve the 
skills and knowledge of persons involved in research and development 
roles who previously received only minimal training, and (3) meet the 
needs of researchers outside the disciplines of education and psychology 
who are increasingly turning their attention to the improvement of 
educational quality. The intent of this project, however, was not to 
enter in competition with institutions or organizations that can best 
perform certain training functions, but rather to continue and expand 
the training activities which a professional association is best able 
to operate. 

Four alternative training formats or models were selected as 
being especially relevant to achieve the objectives of the grant. They 
were: traveling training institutes, intensive presession courses, annual 
meeting training activities and the development of instructional packages* 
Each of these models is described in the following sections of this report. 
Evaluation data for each model, and for the project as a whole, are 
reported in a separate section. 

As a major initial step toward implementation of the grant the 
Association's Standing Committee on Research Training (an outgrowth of 
the 1969-71 AERA Task Force on Training Educational Research and Research- 
Related Personnel) convened in Los Angeles on Februarv 20-21, 1972. 
The committee was composed of W. James Popham, University of California, 
Los Angeles, Chairman; Jason Millman, Cornell University; Blaine Worthen, 



University of Colorado; Robert Morgan, Florida State University; Di; . id 
Merrill, Brigham Young University; Frank Farley, University of Wisconsin; 
and Fran Byers, Student Representative. 

The chiei: concern of the Committee during that meeting was t^e 
development of an organizational structure, irrespective of specific 
personnel, that would produce an active, ongoing training enterprise. 
After extensive deliberations, a unique relationship between the 
specialists on AERA's Central Office staff and the representatives of 
the educational research community who served on the Research Training 
Committee was agreed upon. This organizational structure incorporated 
and capitalized on the expertise of the individuals involved, with 
a member of the Committee directi.-*^ each of the four training models. 
The specific assignments were as f olll:)ws i 

Model A - Traveling Training Institutes, Jim Mitchell 
Model B - Intensive Pre- or Post-Session Courses, Frank Farley 
Model C - Annual Meeting Training Activities, Jay Millman 
Model D - Development of Instructional Packages, Dave Merrill 
Evaluation - Models A,B,C, and the follow-up, Blaine Worthen 
The director of each model assumed the responsibility for its 
budget, time schedule, implementation, and final report. Although 
the individual directors of the four models were given considerable 
latitude in the implementation and operation of their model, the 
Committee retained responsibility for overseeing the planning, design, 
operation, evaluation and dissemination of the objectives and/or products 
of the project. In this regard, the Committee met on six different 



occasions during the period of the grant. The overall coordination of 
the study was the responsibility of the principal investigator, Richard 
A, Dershimer, Executive Officer of AERA, Administrative supervision was 
furnished by AERA's Deputy Executive Officer and co-director of the projec 
William J, Russell. Conceptual and substantive supervision was the respon 
sibility of the Chairman of the Research Training Committee and co-direct- 
or of the project, W. James Popham. 

The following four sections of the report detail the rationale, 
objectives, organization and activities of each Model, Section 5 
contains the evaluative data of each Model and for the project as a 
whole. The sections are individually authored by the model's director, 
A summary and the conclusions are contained in the final se,.tion of 
the report. 



Model A: Traveling Training Institutes 
James V. Mitchell, Jr. 
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The Rationale . The rationale underlying Training Model A - Traveling Training 

Institutes - was described as follows in the original proposal: 

The record of success enjoyed by AERA*s Research Training Sessions at its 
annual meeting suggests that replication of this format may be the optimum 
model for transmitting certain skills and knowledge. However, the limited 
number of participants who find it possible to attend the Annual Meeting, or 
who can be accommodated by such training sessions creates a problem which 
finds its most logical solution in Model A of this proposal. Specifically, 
the; formation of a traveling research training institute is proposed in which 
a particular course directed by scholars with a particular expertise, will 
be available in different geographical locations three or four times a year. 

It is clearly evident that this model will significantly increase the oppor- 
tunity and facility with which a number of individuals could update their 
research, development, or evaluation skills. Prior evidence indicates that 
training offered in such a manner would be widely received among research 
and research-related personnel in the field who are unable to attend one of 
the annual presessions. Moreover, in this era of shrinking educational budgets 
it is increasingly necessary to reduce the cost of instruction to individuals . 
Regional training sessions represent one mechanism by which this objective 
can be accomplished. 

The determination of the validity and utility of the program that was generated 
from this rationale must be made principally on the basis of two sources of infor- 
mation: (1) a description of how this rationale was iit^lementad in practice - 
the plans that were made and the procedures that were followed, and (2) the evalua- 
tion data that v/ere secured about the several Institutes that were offered in the 
program. The present section does not consider the results of the evaluation 
studies, which will be discussed in a later section, but concentrates instead 
on the plans and choices that were made and the procedures which were followed. 

This section of the report, then, will consider: (1) criteria for the selec- 
tion of institute topics; (2) procedures for choosing the topics and personnel 
for the institutes ; (3) coordination between the Research Training Committee and 
the AERA central office and between the director of Model A and the AERA central 
office; and (4) decisions that were made with respect to how evaluation of the insti- 



tute was to be conducted. 



Criteria for the Selection of Topics , In choosing topics for the institutes 
there were certain criteria or guidelines that served to focus the committee's thinkin 
(1) Since the proposal was concerned with the development of "Training Models" for 
educational research, the institutes should serve a training function and should be 
oriented toward skill development of some kind; participcuits should be able to do 
something as a result of their trciining that they could not do before or could not 
do at the same level of competency; (2) The content of the institute should be re- 
levant to the present needs and trends of the field of educational research and the 
people within it; and the relationship of that content to a current training need 
should be demonstrable? (3) Other things being equal, the content of such institutes 
should be of general, not esoteric interest, and the drawing power of an institute 
with such content should be great enou^ to justify the considerable planning, efforti 
and expense involved. 

Procedures for Choosing Topics and Personnel > There was general agreement, 
at the very beginning of tlie program, that it would be the entire membership of the 
Research Training Committee that would apply these criteria and select the topics 
and instructors for the institutes. Historically, there seemed to have been two 
procedures that were used in making these choices . During the first phase the 
Committee discussed at length possible topics and instructors and ultimately agreed 
that certain well-known professionals should be extended invitations to conduct 
institutes in certain generally defined areas? this approach resulted in invitations 
extended to Michael Scriven and Daniel Stufflebeam to offer an institute on eval- 
uation cind to Melvin Novick to offer an institute on Bayesian statistics. The 
Scriven-Stuf flebean institute was called "Alternative Conceptions of Evaluation" 
and the Novick institute was "Bayesian Statistics and Interactive Corc^uting Systems." 
A conscious attempt was made to secure professionals of some considerable reputation 
who were acknowledged leaders in their fields, would have great di awing power, and 
who could be counted upon to do justice to their topics. Such people, however, tend 
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to be very very busy, and although our first efforts to get the people we wanted 
were attended by no small measure of success, our subsequent efforts to get the 
"stars" of our field were not always as successful. 

This led to the second phase of the selection procedures. It was decided by 
committee members that the second group of topics and instructors would be selected 
on a competitive basis, rather than extending invitations to people designated be- 
forehand. Specifically, it was agreed that the central office would send letters 
to directors of successful previous AERA presessions and invite them to submit a 
proposal for a traveling research training institute. T"- 3 Research Training Comm- 
ittee would then examine carefully the proposals submitted and decide which should 
be funded. This procedure resulted in the siabmission of some excellent proposals, 
and the committee elected to fund several of the best. The coir5)lete list of insti- 
tutes that were offered in the porgram is presented later. 

In retrospect it seems that both of these procedures were defensible ones, and 
especially in the order of their occurrence for this program. Obtaining the services 
of the most reputable men the field has to offer, is worth the effort if it only re- 
sults in the meeting of a few such luminaries. Selecting others on a competitive 
basis from a pool of demonstrably successful presession directors seems equally 
effective as a means of obtaining the services of very, able and talented people 
who may not (or not yet) have the social visibility of the "stars" but who have 
very important contributions to make . 

PxOlationship of the Central Office to the Committee and Director, Throughout 
the project the coordination of efforts between the AERA central office and the 
Research Training Committee and between the central office and the various direc- 
tors of the training "models" was considered to be a most important aspect of the 
experimentation, both conceptual and organizational, that was an inherent part of 
the total program • The Research Training Commj.ttee was most fortunate in having 

the services of Dr, William Russell of the AERA central office to provide the 
administrative and organizational services that were essential to the success 
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of the project. Dr, Russell was also very sensitive to the role conceptions and 
various attitudes that can exist in any setting defining a professional membership- 
central office relationship, and this, combined with his facile. ty for easy working 
relationships, assured his effectiveness. 

Early in the project it was decided by the committee that each of the training 
'•models" would have a director, and that tlie director would be a member of the 
committee. The committee would make general decisions on policy, the director of 
the training model would attenpt to implement these policy decisions in his area, 
making those related judgments and decisions, mostly administrative, that were 
necessary to implement those decisions effectively, cind the central office re- 
presentative would take the organizational and administrative steps to make the 
program operational r The director of each training model was supposed to consult 
wil±i the chairman of the Research Committee on any matters considered to be policy, 
and with the central office representative on any matters considered essentially 
administrative. Since the relationship between policy and administration was a 
close and interlocking one, in practice it was the committee chairman, the director, 
and the central office representative who often consulted togetJier and made some of 
the decisions necessary to implement policy. 

In the case of the Traveling Research Training Institutes the director's role 
had some features of a "middleman" role. Once the committee had made certain de- 
cisions about topics and instructors, it was his responsibility to contact those 
chosen to explain the program and extend an invitation to participate. If 
those chosen could not participate, or if other problems arose, the director con- 
sulted with the committee chairman and the central office representative, or, in 
certain instances, returned to the committee for further direction. These organx- 
zational relationships worked reasonably well and could be recommended to others , 
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especially if there was concern on the part of an organizational luembership that 
control and implementation of policy reside securely with the membership. If this 
is not a matter of great moment^ however, it is probably simpler organizationally, 
more economical of time, and more facilitative of coordination, if the functions 
of the director's * middleman" role are assirrned to the central office. 

Procedures for Evaluation. Evalu ion was considered to be an integral part 
of the program from its inception, and the only question was how it was to be eff- 
ected. Originally it was conceived that the directors of the training models would 
have major responsibility for the evaluation of their area, with consultant help 
from evaluation specialists, but later in program development it was felt that it 
would be best to have an evaluation specialist conduct the evaluations of all four 
training models, with input from directors as required. Dr, Blaine Worthen of the 
Research Training Committee, an acknowledged expert in evaluation, was asked by the 
committee to assume this r< 

In the case of the Tr>3v'eling Research Training Institutes, it was decided that 
both formative and summative evaluation procedures should be applied to the institutes, 
and that each session of an institute should have an on site observer who could pro- 
vide immediate feedback to the instructor (s) and input for the formative evaluation 
procedures. Since there were two or three sessions for each institute, earlier 
evaluative results could also be regarded as formative data for later presentations 
of the institute. Each session of each institute was sxabject to a thoroughgoing 
evaluative analysis, and the results of these evaluations were presented in report 
form. A summary of evaluation results is presented in another section of this final 
report. 

Descriptive Data on the Institutes Offered. A listing of the sessions of 
all institutes offered, the dates and places of these sessions, and the instructors 
is included in the evaluation section of the report and in the appendix. 
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MODEL B: ANNUAL MEETING PRESESSIONS 
Frank H. Farley 

AERA's considerable history of offering presessions and postsessions 
at its annual meetings indicated that successful as the sessions were, 
they largely neglected audiences often in need of training in educational 
research. Therefore, the intent of Model B was to make a concentrated 
attempt to reach individuals beyond the stereotyped AERA member and 
audiences beyond those which normally participate in AERA sponsored training 
sessions. Thus the new presessions, held in conjunction with the 
Association's annual meeting, were to be specifically designed for 
audiences very broadly defined as educational researchers. 

Proposals were solicited for this model from prospective directors by 
mechanisms identical to those used for the Association's traditional pre- 
session-postsession program. The criteria used by the committee in 
selecting proposals consisted of: (a) demonstrated competency of the 
director(s) and instructional staff; (b) relevance of course content; (c) 
development of explicit objectives; (d) instructional organization of the 
session; and (e) expected appeal to a wide and relevant audience. On the 
basis of the proposals received, the committee selected two training 
sessions to initiate Model B. They were: "Research on Reading Acquisition: 
With an Emphasis on Deprived Populations" directed by S. Jay Samuels, 
University of Minnesota, and "Computer Managed Instruction" directed by 
Frank B, Baker, University of Wisconsin. Descriptions of these sessions 
are included in the appendix. Unfortunately^ the latter session was 
cancelled due to an insufficient number of advance registrants. 

The sessions were publicized in the same manner as the traditional 



11 



presession.s ond postsessions that wero olTcre.^. at the annual .nect inj^ in 
New Orleans. The registration fee schedule of $45 per .session day for 
AERA members and $50 per session day for non-AER/i members, was assessed 
participants of both the Model B session and the traditional presessions 
and postsessions. 

Although the presession that was offered in this model v;as generally 
considered successful , (a detailed evaluation of the session is included 
in a subsequent section of this report) the model in general was not 
viewed as achieving its full potential. Upon reflection about this model 
one can speculate about a number of variables that may explain the less 
than enthusiastic support or participation that the sessions received. 
However, one of the simplest and most logical explanation that has been 
advanced in retrospect is that considering the sessions were directed 
principally at broad and new audiences of educational researchers who do 
not necessarily attend AERA's Annual Meeting, offering the sessions in 
conjunction with AERA's Annual Meeting was not as positive an inducement 
as it would have been if the sessions were conducted at the professional 
meeting of other relevant organizations. Such a hypothesis obviously 
doesn't preempt other variables (fee schedule, publicity mechanisms , appeal 
or relevance of the selected sessions, etc.) from consideration of the 
overall efficacy of this model. Such comments and observations, however, 
are reserved for a later section of the report. 
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rodel C; Annual Meeting Training Acti\ ities 
Jason Millman 



Characteristically, scicnti :"ic, professional society rn-j^j tir.'^:. 
are devoted to research reporting, inspirational messages, socia], inter- 
actions and the like. Training of professionals is seldom consciously 
performed at such meetings. The purpose of Mcdel C of this grant was to 
examine whether successful training could be conducted during th^ annual 
AERA meeting. 

The annual meeting was seen as an especially suitable time for 
such training activities for several interrelated reasons. First, a 
large number of potential trainees would already be at the training site 
by virtue of their attendance at the annual meeting. Second, the time 
of the annual meeting permits maximum utilization of the "human library" 
resource, i.e., the scholar who attends or participates in the annual 
meeting. ^The most able scholars are frequently the most busy and might 
not be available to provide such training activities at other times. 
Third, potential trainees are also busy and might not be able, for 
example, to take off the additional time if training activities were 
conducted sequentially (but not concurrently) with the annual meeting. 

There exists a major drawback to holding training activities 
during the annual meeting. The annual meeting is a busy uime for most 
participants, for there are already a multitude of activities competing 
for their time. It is for this reason that the three training variations 
conducted at the anr^ual meeting were designed to be (a) physically 
accessible to the participants, (b) of relatively short duration, and 
(c) available either at several times or during times thought less likely 
to conflict with other attractions. 



Three variations were chosen: (a) minicourses , (b) conver- 
sational hours, and (c) exhibits of self instructional materials. Each 
of these variations will be described under the headings: rationale/ 
solicitatioii and selection of trainers or products, and publicity. The 
evaluation of these efforts, however, is contained in a later section 
of this report. 

MXNICOURSES 

Rationale 

For many years AERA has engaged in offering training courses 
of two to five days. Attendance at such sesjsions speak to their accept- 
ability. It seemed only natural, therefore, to provide such a mode of 
training during the annual meeting — shortened, of course, to a few 
hours in length. 

There was concern, however, among members of the Research 
Training Committee that such short courses might be hard to distin- 
guish from information presentations offered as part of the regular 
annual meeting program. Partially for this reason, and partially 
because of the objective of this research grant, great emphasis was 
placed on the importance of skill improvement. This emphasis was 
coimnunicated in the call for proposals. 

The purpose of the sessions will be to 
transmit specific skills in research, 
development, and evaluation to partici- 
pants rather than to carry on a general 
discussion. . . . Individuals interested 
in serving as instructors should pre- 
pare a short statement specifying what 
the participants will be able to do by 
virtue of attending the instructional 
session. The intended skills to be 
developed should be identified in as 
specific terms as possible. Educational 
Researcher , May 1972, p. 19. 



Solicitation and Selection of the Minicourses 



Twenty-four proposals were received by Model C director, 
Jason Millman, by the deadline date of July 15, 1972. All 24 proposals 
were circulated by mail and rated independently by seven members of the 
Research Training Committee according to the following scale. 

1. I think this proposed minicourse should 
be given serious consideration. 

2. I'm not sure about this minicourse and 
would be willing to be persuaded either 
way (in or out) . 

3. My vote on this proposed minicourse is 
NO. 

The Committet: members met on July 27 and 28 in Chicago and, after due 
consideration of the criteria, selected the four proposals with the 
highest ratings. 

Criteria for selection explicitly agreed upon by the Committee 
members were: (a) expected draw (the minicourses were intended to be 
self-supporting), (b) potential for developing specific skills, (c) 
identification of skills, (d) balance with other offerings, and (e) 
avoidance of overlap with presession offerings. In addition, other 
criteria included: (f) importance of content, (g) feasibility of the 
skills being acquired in the time allotted, (h) potential quality of 
instruction, and (i) competence of the instructor (s) . 

A description of the selected minicourses is provided in 
Appendix C . The course by Leonard Marascuilo was subsequently can- 
celled because of the illness of Professor Marascuilo at the time 
the course was scheduled to be held. 
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Publicity 

A call for minicourse proposals was issued in the May 1972 
issue of Educational Researcher. (See Appendix C ) In addition, let-- 
ters were sent to a large number of previous directors of AERA training 
sessions encouraging them to submit proposals for mini courses . Reminders 
of the proposal deadline appeared in the June and July issues of Educational 
Researcher. 

Once the minicourses were selected, descriptions appeared in 
the Educational Researcher (see Appendix c ) ^ in the Annual Meeting 
Program, and in a flyer widely circulated at the convention (see Appendix 
C ). 

CONVERSATIONAL HOURS 

Rationale 

It was believed by the Research Training Committee that one 
of the most valuable ways young researchers could improve their re- 
search would be to discuss their proposed research efforts with promi- 
nent scholars in their respective areas of research. In spite of the 
potential advantage to the few individuals who would be recipients of 
such "tutoring", such one-on-one training was seen as neither feasible 
nor efficient. Consequently, it was decided that group conversations 
would take place between the prominent scholar and the less established 
researchers. Further, in an effort to simulate some of the characteris- 
tics of a tutoring session, the less established researchers were en- 
cour=iqed to submit directly to the senior researcher, prior to the 
annujil meetin^^, a one- or two-page letter indicating the topics they 
would like discussed. iijo'^ Appendix C ). 



The conversational hours were designed to differ from graduate 
students seminars in two ways: the conversational hours were open to all 
participants at the annual meeting, not just graduate students, and, 
more importantly, it was intended that the discussion would be focused 
on the specific concerns previously communicated to the conversational 
hour leaders rather than on general issues. 
Solicitation and Selection of Conversational Leaders 

It was originally intended to select the conversational hour 
leaders from among the more frequently mentioned researchers nominated 
on special forms printed in the Educational Researcher , (See Appendix 
C) . However, only four nominations were received. Therefore, in con- 
sultation with Committee chairman, Popham, Model C director Millman 
invited four individuals having high visibility in AERA. (Appendix c 
contains an example of the letter of invitation and charge to the 
conversational leaders.) The four individuals contacted agreed to 
participate. Their names and areas of expertise to be discussed are 
listed in Appendix C . 
Publicity 

Nominations for conversational hour leaders were solicited in 
the May, 1972 issue of Educational Researcher (Appendix C ) - Once the 
leaders were selected, announcements appeared in the Educational Researcher 
(Appendix C , in the Annual Meeting Program, and in a flyer widely circula- 
ted at the convention (see Appendix C ) . 



TRAINING EXHIBITS 

Rationale 

Largely due to Federal support, R-D'-E- training materials were 
becoming available, but still were relatively unknown to the educational 
researcher. Although it was hoped that convention goers would spend some 
time going through the training materials during the annual meeting, it 
was anticipated that the lengt.h of time needed for such an activity pre- 
cluded use of the materials in the manner intended. 

Nevertheless, the exhibits were seen as valuable for the pur- 
pose of publicizing their availability and possible adoption for later 
classroom of self use applications. For this reason, sponsors of the 
training materials were urged to provide flyers describing the product 
and informing interested persons how to receive more information or how 
the materials could be purchased. 

At first, it was intended to charge a $1.00 fee to participants 
who attended the "study center" (see Appendix c ) / t)ut this was dropped 
and no fee was assessed. A centrally located room in the headquaters 
hotels made available in order to encourage maximxom utilizations of the 
study center, was kept open for several hours each day of the annual 
meeting. 

Solicitation and Selection of r ^xhibits 

A letter (see Appendix c ) was sent to the roughly two dozen 
directors of projects sponsored by the Research Training Branch of the 
USOE (now NIE) . Additional letters were sent to approximately six 
individuals known to have produced training materials . 
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With one exception, all developers responding positively to 
the solicitation were invited to participate in the exhi±)ition. ,A 
listing of exhibitors and their products or materials are listed in 
Appendix C ). The one exception involved a developer who, after consulta- 
tion with Model C director, Millman, agreed that his product was not a 
training product and withdrew his request to participate. 

A list of Exhibitors and their Training products or materials 
are lis ed in Appendix C - 
Publicity 

An announcement of the availability of the exhibits appeared 
in the Educational Researcher (Appendix c ) f in the Annual Meeting 
Program, and in a flyer widely circulated at the convention. (see Appendix 
C ). 



Model D: Development of Instruetlonal Packages 



David Merrill 



I. THE CHARGE 

a. lu early L972, AERA submitted a proposal to the U,S. Office of 
Education requesting funds for exploring various training models. One of 
those approaches was described as follows: 

Model D - Development of Instructional Packages 

Any formal "live" instructional course is severely limited by the 
impossibility of disseminating it to many learners without the 
presence of participating instructors. To remedy this deficiency, 
training Model D is proposed to test the possibility of preparing 
exportable training materials. 

In examining alternative approaches in the development of in- 
structional materials, it is apparent that there is a clearly defined 
continuum of both cost and validation. At the one extreme are the low cost, 
unvalidated test book materials; at the other extreme are the costly, 
heavily field-tested, carefully validated instructional materials 
currently being produced, for example, at the USOE-supported regional 
laboratories. Part of this proposal's Model D involves a middle 
range of expenditures and validation which capitalizes on AERA*s 
wide experience in conducting and evaluating instructional training 
sessions. 

Commencing with the currently scheduled April 1972 training 
sessions , several session directors would be invited to identify and 
propose for filming one or more training segments , which seem par- 
ticularly suitable for export to a larger audience. A review panel 
from the Research Training Committee would evaluate the potential 
of the resulting proposals in terms of need, appeal, self-contained 
qualities (i.e. , no special prerequisites) and suitability of in- 
structional plan. Two such training sessions would then be scheduled, 
each involving perhaps two or three discreet 4 5-60 minute training 
segments. The directors would subsequently be advised to setup 
a field test version of the selected segment prior to April, 1972. 



Before these field tests are actually conducted, the Research 
Training Committee will assume the responsibility of reviewing 
the nature of the training segments, the competencies they were 
designed to produce, and the means by which they could be 
evaluated. At the time of the field test, two or more members of 
the training committee would be present to monitor the effort 
and to aid in the formative evaluations of the instructional segments. 
The training participants in the field test would offer suggestions 
for program modifications, and in addition, their performance in 
competency measures would aid in the formative evaluation. 

After making changes indicated by the field test, the training 
director and his staff would, in consultation with members of the 
Research Training Committee, select an appropriate recording medluir.: 
eight, sixteen, or thirty-five mm film. Arrangements would then be 
made for technical support services appropriate to the designated 
training session segments. In some Instances this might involve trans- 
porting the participants to a studio; in other cases, the recordings 
could be made under normal meeting circumstances with augmented 
local facilities. 

After a product evaluation by the Research Training Committee, 
dissemination channels, both internal and external to AERA, would 
be employed to bring the product to a wide audience. It is important 
to note that only two sessions are to be designated for 1972. The 
experience gained in this venture will undoubtedly lead to an im- 
provement in the model for the 1973 February AERA Annual Sessions. 
It is anticipated that at least five to ten hours of exportable in- 
structional materials will be prepared. Consistent with the 
expectation of making research training a self-supporting enterprise, 
a fee will be assessed for rental or purchase of the instructional 
materials. In addition to the substantive evaluations of the model, 
cost accounting of production and marketing will allow financial 
evaluations of the instructional materials. 

Another element of this model might be development of supplementary 
audiovisual materials to accompany live presentations of training 
sessions; obviously many sessions would be of much greater in- 
structional value if augmented with multimedia products. 

Model D reflects AERA's continued commitment to explore 
ways in which research training can most effectively be conducted. 
Its entry into the more sophisticated modes of media presentation is 
a logical extension of its successful activity in the area of cassette 
Instruction. 



b. At the March 1972 meeting of the Standing Committee on Research 
Training, the various possibilities for implementing this charge from the 
proposal were discussed. The original intent was to make videotapes of 
some of the pre and post-sessions at the convention and tc offer these 
videotapes for sale to irterested researchers. After some discussion, it 
was decided to attempt a full-blown instructional development effort in 
which validated instructional packages were produced for commercial 
distribution under the auspices of AERA. It was this second approach which 
was agreed to by the committee. 

c. Because of M. David Merrill's membership on the committee, and 
the fact that Brigham Young University had recently established a major 
instructional development division ^ it was decided that he should assume 
responsibility for his institution in accepting the contract to produce two 
research training packages. Because this activity was viewed as a pro- 
fessional opportunity, it was agreed that no overhead would be charged by 
the university and that the packages would be produced on a cost basis 
with no fee. (Appendix A is a copy of the agreement signed by AERA and 
BYU) 

d. After considerable discussion, it was decided that one of the 

mo^L successful presessions was that offered by Joe Ward and his associates 
on linear regression models. Dr. Ward had previously expressed an interest 
in producing a training package on "Applied Decision Making" for adminis- 
trators. It was decided to approach Dr, Ward and explore the possibility 



of his becoming associated as an author with this project to produce such 
a package. M. David Merrill was given the assignment to contact Dr. 
Ward and explore his willingness to serve in this role. This contact was 
made shortly following the Research Training Committee meeting. Dr. Ward 
was enthusiastic about the project and agreed to participate for a modest 
honorarium and travel expenses. 
II . THE PLAN FOR BYU 

a. Phase I , The project actually got underway in late May of 1972. 
An advanced graduate stuaent was selected as project director. Contact 
was made with Dr. Ward and arrangements were made for him to visit the 
BYU campus. He visited the campus and presented a lecture to the faculty 
and students of the IR&D Department. This lecture was videotaped for 
future reference. Dr. Ward also distributed notes that could be used In 
preparing the content. The project director was to make an outline from 
the videotapes which was then to be presented to Dr. Ward for his critique. 
A PhD level faculty member of the Department of Instructional Research and 
Development was assigned during this Phase I period to serve as an in- 
structional psychologist, consultant, and advisor to the project. 

During this phase, the operation at BYU was in what we have come to call 
"shoe shop" mode. That is, rather than a team of people working on a 
project, projects were typically assigned to one faculty member, perhaps 
assisted by a student. In this case, the graduate student project director 
and faculty instructional psychology advisor were assigned to carry through 



with the project. Their task for the summer was to do a content analysis of 
the material presented by Dr. Ward to develop a formal set of content 
definitions and examples which could be used in the subsequent development 
of a training package. A set of materials was submitted to Dr. Ward for 
his approval in early September. 

b. Phase II . During the late summer of 1972, the faculty in the IR&D 
Department at BYU seriously discussed going to a team approach for in- 
structional development. The personnel in the department were divided 
into four teams. These teams included an Instructional design team, an 
evaluation team, an implementation team, and a product development team. 
In addition to these four teams , a given project always Included a subject 
matter author and various packaging specialists represented throughout 
the Division of Instructional Services at BYU such as the Press, Instructional 
Television, Motion Pictures, and Instructional Photographies. 

Because the AERA project was in its initial stages , it was seen as an 
excellent vehicle for implementing the team approach. On September 1, 
therefore, the project was reorganized into a development team. The graduate 
student originally appointed remained as the project coordinator whose 
assignment was to shepherd the project through the various team groups. 
The teams consisted of the PhD level research associates and graduate 
student research interns, organized as indicated in Table 1. Figure 1 
Illustrates the planning pert which was used for this project, together with 
the estimated mandays required for the project. It will be noted that the 



project is divided into various phases and each of these phases has been 
assigned to a team. The graduate student project director was to be the 
overall project coordinator to assure the adequate communication between 
teams. Each of the teams were to contribute their relative contribution 
to the ultimate production of the package. Appendix B contains a somewhat 
more detailed outline of the original team organization and tlie description 
of the function of the various groups. According to the pert, the project should 
have been finished, ready for evaluation, in time for presentation at the 
convention in late February. 

Insert Table 1 & Figure 1 about here 

III . THE EXECUTION OF THE PLAN 

a. The top half of Figure 2 is the pert chart translated onto a timeline 
with the due dates indicated at the end of the arrows. Each o* the major 
steps in the team development model are indicated. The starting time for 
the beginning of that step is indicated by the beginning of the arrow. The 
content analysis and content specification stage obviously went back into 
the summer, since that was the effort that was conducted mainly during 
the "shoe shop" phase of the project. 

The bottom half of Figure ?- indicates the actual execution of the plan. 
There are some noticeable problems in the execution. The first problem is 
that no formal content specification was ever produced. There was a great 
deal of content material written up. Most of it, however, remained In the 
form of notes and relatively unorganized material prepared by the project 



director and the faculty instructional psychologist. The material was 
critiqued by Br. Ward and much of it approved by him, but it was never 
organized into a formal document. Therefore, this content analysis is not 
available for inclusion in this report. Nevertheless, it should be noted that 
with the introduction of the formal team model, the initial products (the needs 
and goals document and the systems design document) were produced as 
scheduled. These constitute Appendix C and Appendix D of this report. 
Following the production of the systems design document, a version I 
prototype mockup was produced by December 1. This mockup constitutes 
Appendix E. This document is in extremely rough form since it is only a 
mockup. It was submitted to the evaluation group and the implementation 
group for critique in the early part of December. In addition, it was 
submitted to the Photographies Department and to the Press for suggestions 
concerning packaging . 

During the month of December and early January there was a breakdown 
in the team approach. The cause of this breakdown will be discussed during 
the critique. Nothing more was done with the version I prototype mockup. 

The program went into crisis mode in the early part of January^ and the 
package which was produced and eventually presented at the AERA convention 
was produced with a combination of the production group and an off -campus 
group who specialized in the production of educational simulation games. 
The version II prototype mockup which was presented at tho convention is 
presented here as Appendix F of this report. 



It should be noted that because of this breakdown in the team approach 
no formal evaluation design was produced and no formal implementation 
design was produced. Work and discussion was carried on in these areas, 
but no formal documents, as required by the model, were ever produced. 
Because no evaluation design was produced and because the project did 
break dgwn, the formative evaluation indicated above as item 10 in Figure 2 
was never conducted. 

The summative evaluation had originally been planned to be conducted 
by Blaine Worthen at the University of Colorado. While he was standing by 
and ready to perform such an evaluation, the package was not ready on 
time and no summative evaluation was ever conducted. 

Table 2 presents vita! information relative to each of the primary 
development activities in Figure 1. This indicates the activity, the product 
which was produced as a result of that activity, the personnel involved 
in that production, the manpower loading both in estimated mandays and 
actual mandays, and the cost for each activity. These costs are the 
costs up to the presentation of the material at the convention in February. 
An accurate audit trail was possible because each of the personnel working 
on the project was required to keep hours, so that mandays are actual 
time spent on the project and not estimates « This makes the cost accounting 
extremely accurate. There were some additional mandays on the part of the 
Implementation and evaluation groups which were charged to the project, but 
since no Intermediate product was produced, there were not Included in Table 2. 



IV. THE CRITIQUE OF THE PLAN 

a. "They're off — slowly," The initiation of the project did not 
proceed as expiditlously as might be hoped. The initial contact with Dr. 
Ward and his initial visit to BYU was very successful. There was con- 
siderable effort by the project director following this period in time to 

put together some formal content materials. Dr. Ward made a second visit 
to the campus, reviewed these materials, and discussed the progress on 
the project. In the late part of the summer^ a faculty-level instructional 
psychologist assumed the assignment to monitor and advise on the project. 
While he spent a number of days in discussion on the project, no formal 
material was produced by his effort other than some initial critique of the 
content materials. 11 should be noted that there was some concern expressed 
at that early period that perhaps the content material was not as strong as 
it could be. However, hope was expressed that with additional work, 
adequate examples could be found and that the material could be organized 
in such a way to present a meaningful package, especially to those 
administrators who had very little research training. This was the Intended 
audience of the product, therefore , it was felt it would still be a very 
profitable enterprise. 

b. The team approach — let's make it an example . When the team, 
development model was implemented September 1, there was a great deal 
of enthusiasm on the part of all of the people involved at BYU. It was 
determined at this time to make the AERA project an example of the team approac 



since many of the other projects In which we were involved were in various 
stages of development and the AERA project was relatively new, it seemed 
like a perfect example to push through the entire process. As evidenced by 
the needs goals^ systems design, and the version I prototype, considerable 
progress was made during this period and the team approach worked 
according to plan. While the prototype version I is extremely rough, it is 
evident that there was still a great deal of promise , but that a great deal 
of work needed to be done to improve the content to find more adequate 
examples. As of December 1 it appeared, in hindsight, that a reasonable 
and marketable product would still result, 

c . The cold winds of December, When the version I prototype mockup 
was prepared, it was presented to the evaluation group and the implementation 
group for their use in preparing evaluation plans and implementation plans. 
Obviously, their first task was to review the product at some length. Their 
reviews were very critical. At this time (December, 1972) some very strong 
criticisjn was leveled by tlie head ox the evaluation group and the head of 
the implementation group, questioning whether there was enough content 
in the product to make it a worthwhile enterprise. These criticisms were 
submitted to Dr. Ward who made another visit at this time. His own comment 
was that perhaps the content was a little weak and he wondered if the product 
had sufficient value to warrant further production. The general feeling 
following this criticism was that although the criticism was rather severe, 
the product still should be produced, and while it would probably not be the 



most outstanding package in the world, it would still have validity and 
usefulness for a large number of the AERA audience, especially those who 
are in administration and who are responsible to get research started. 

At this point, it is probably advisable to explore the sociology of 
criticism. The project director, while a very competent student and now 
a very qualified instructional developer, was not an extremely aggressive 
person by personality. This coupled with the fact that he was a graduate 
student and therefore subject to constraints on all graduate students of 
pleasing his professors, made him especially susceptable to the criticism 
that was solicited at this particular time. The director of development 
who produced the prototype package had been on the faculty only three 
months . He was the youngest of the faculty members , anxious to find his 
place , and not anxious to raise waves . The head of the evaluation groups 
was a substantial faculty member of considerable reputation, well 
admired by both the development group head and the graduate student 
project director. Obviously his criticism would be taken very seriously, 
especially since he was the faculty membcjr involved in the project earlier 
in the summer. The head of the implementation group was also a professor 
of considerable stature, with a history of extremely successful development 
products. His criticism also would be taken relatively seriously, especially 
by a graduate student and a new faculty member. This unfortunate com.- 
bination of personnel probably increased the devastating effect of the 
December criticism more than was warranted. That is, had the project director 



been a stronger person relative to those offering the criticism ^ he probably 
could have sifted for those things which would have improved the project 
and eliminated those items of criticism which were not relevant to the project. 
The same is true of the head of the instructional development group. As it 
turned out, however, because of the difference perceived in stature, the 
criticism was taken very seriously and there were considerable doubts 
raised about the value of the team process. This obviously was quite un- 
related to the criticism^ but it did cause considerable disruption of the 
entire project. The period of December and early J-^nuary therefore was one 
of relative chaos in regard to this project. 

A complicating factor was a tremendous crunch from other work. The 
TICCIT project (a large National Science Foundation-funded computer-assisted 
Instruction project) was taking a considerable amount of effort at this 
particular time, and those who could have put their efforts into solving the 
problem on AERA were diverted because of this tremendous pressure from other 
projects. In addition, a number of other projects were also in crisis mode 
at the same time. Therefore, the head of the development group merely 
turned his attention to other more pressing matters and waited for the 
criticism to be resolved. 

Dr. Merrill in his role as project monitor was not aware of this particular 
sequence of events as he should have been (consequently he did not take 
steps to provide the needed resolution of the conflict). The criticism, 
therefore^ was much more devastating than it should have been, and what 



should have turned out to be a relatively good project was for all practical 
purposes aborted at this point In time. 

d. The crisis crash, Tanuary-February 1973 > In mid January it became 
apparent that progress on AERA version I was at a virtual stantstill, and 
that the deadline of the convention was rapidly approaching. It was felt 
that some product must be provided. A decision was made that since we 
would be unable to get a validated product to the convention on time, that 
at least we should provide a prototype. The development group was called 
back and various alternatives were explored. At this time a group, un- 
related to IR&D, who specialized in the production of educational games , 
had contacted the department looking for opportunities to be Involved. 
Because of their experience , it looked like they might be able to provide 
some valuable Input to the AERA project. They were hired to provide some 
simulation exercises with some of the materials developed during the summer. 

After the simulation people had finished their work in providing some 
programmed exercises, these were packaged together by the production 
group into the version of the material that was presented at the convention. 
It was decided that what the material lacked in content should be made up 
for in pazaaz, therefore a relatively fun package was developed, but the 
content was certainly less than desirable. 

e. The criticism of the committee. The committee's review of the 
project was relatively kind, but serious questions concerning the adequacy 
of the content were raised. These comments constitute Appendix G of this 



report. It was obvious that what had been produced was not adequate, and 
would not meet the needs. It was therefore decided that a careful evaluation 
should be done and three alternatives should be approached. The Research 
Training Committee was polled concerning these alternatives: 

1) Abort version I , go on to the second package. 

2) Rwise version I sufficient to meet the contract demands from the 
Office of Education. 

3) Revise the project sufficient to produce a distributable package, 
f . Dr. Green, who had considerable experience at Indiana with Ivor 

Davies , was convinced that a substantial amount of material existed on 
decision making, and that were some of this material to be gathered together, 
a much more substantive package could be put together. He was therefore 
authorized on BYU money to go to Indiana, interview Ivor Davies, and 
gather together a considerable set of this material, During the March, 
April, and May period. Dr. Green did a great deal of work in gathering to- 
gether materials on decision making. These materials then were put 
together in a version III oackage, which is also appended (Appendix H) to 
this report. It was decided, however, in May that this package, while 
considerably better in substance and content, was still not an instructional 
package, and that it would perhaps be better to go on to the second in- 
structional development package rather than to continue to save the "Applied 
Decision Making package. This appeared especially advisable since 
the Research Training Committee members were somewhat hesitant to vote 



for a continuation and seemed to feel a marketable package would not be 

possible. A polling of the BYU faculty also indicated a desire to abort 

the first attempt and go on to the second package. Therefore, it was decided 

that all of our efforts would go into this second instructional development 

package. 

g. The disaster of the summer of '73. The entire instructional develop- 
ment program at BYU was thrown into relative chaos during the summer of 
1973 . A number of precipitating factors were Involved, ?U of which are 
not appropriate to be reviewed here. Among them, however, was the 
promotion of the director of the Instructional Services Division, Darrel 
J. Monson, to an Assistant Academic Vice President. This necessitated 
the reorganization of the division. This reorganization was not resolved 
until September, thereby leaving the leadership of the division in question. 
There was also considerable struggle over the existence of the doctoral 
program in instructional psychology as to whether or not it would continue. 
This left the status of many of our faculty and particularly M. David Merrill, 
in a relatively unstable position for a period of three mondis. In addition, 
there was a tremendous continuing pressure from the TICCIT project , and 
other instructional development projects at BYU. In July, hy prior arrange- 
ments , Dr. Green left the campus for 6 weeks. Progress on the first AEPA 
product virtually stopped at that time. The commitments of other people 
prevented our becoming involved with the second AERA project prior to 
September. 



V. THE PROLOGUE 

In this section of the report it is appropriate to explore probable reasons 
for the death of the AERA instructional development project. The number one 
sweepstake reason is inadequate management. M. David Merrill who had 
ultimate responsibility for the project did not monitor carefully enough the 
progress that was being made. He was not alert to the devastating effect 
of the criticism in the December review, therefore did not follow through 
in pulling the project out of the fire. Most of the failure then must be 
chalked up to inexperience in managing a team approach. Management also 
broke down in that the team approach was under serious question and criticism 
during the spring and there were a nurrbar of faculty members who felt that 
a team approach would not work out after all. All of these factors combined 
to prevent the successful completion of the project. 

A second reason, in order of priority, is related to the comment on the 
sociology of criticism. A very vabable management lesson has been learned 
from this project. It is as follows: While criticism is extremely desirable, 
it is important that those who direct the project have the ability to sift the 
criticism, accept that which is relevant, and reject that which Is not relevant 
so that they can proceed relatively uninhibited. That is, if the director of 
the project, the person who has responsibility for its production, if in a 
status position which is viewed as less powerful than those who are offering 
the criticism and if he does not have a high status backup, he feels that 
his own position is threatened by the criticism. ThJs criticism is therefore 
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likely to have a devastating effect. It is important that criticism be sought 
carefully / that it be adequately screened, and that those who are in the 
management positions have the necessary backing and power to ride the 
waves of criticism and still produce a package. A further outcome is the 
importance of clearly defining the role of the critic as a decision role or 
an advisory role. In this situation, such a distinction was unclear and the 
criticism essentially was Interpreted as decision making rather than advisory. 

The third reason for the death of this project was the lack of a formal 
content specification. In the early summer Dr. Ward had been asked to 
prepare a formal content specification, outlining each of his generalities, 
definitions of concepts and rules , and idantifying for each a set of examples. 
If he had received the primary responsibility for providing this input and 
insuring its adequacy, then there would have been adequate substance from 
which to work. As it was, a graduate student was left with the major task 
of developing most of the content examples. It is not surprising that these 
examples were not as adequate as those that could have been developed by 
a more experienced content expert. This lack of a formal content specification 
proved to be a handicap throughout the project. It was the cause of the 
strong criticism in December that effectively killed the project. 

A fourth problem was the intrinsic weakness of the content. Looking 
back, it is apparent that for Dr. Ward this approach to decision making was 
more of a hobby than his area of real expertise. Had he provided input 
related to multiple linear regress ion and the material presented in his workshop 
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and his pre-sesslon, there could havr? been some extremely strong and 
valuable material available. As it was, his own understanding / his own 
presentation of this area was somewhat superficial, and was not the indepth, 
reasoned, scholarly approach that would have made for an excellent research 
package. It should not be assumed that this insight was available to us - 
or to Dr. Ward during the summer we were working with him. It is only after 
careful analysis and hindsight that this particular problem has come to light. 

A fifth problem was the lack of an on-the-scene subject matter expert. 
It is extremely difficult to produce an adequate Instructional development 
package by long distance, unless there is a considerable definition for the 
remote author as to what his input should be. A remote author is probably 
possible if an adequate content specification is provided by him. Usually, 
however, an onstte author is critical in providing the kind of guidance and 
the kind of validity checks on the content which would make for a strong 
package. 

There are probably a number of other reasons for the demise of this 
project, but these seem to be the most pertinent, based on the autopsy. 

BYU is extremely regretful that this project did not proceed as planned 
and that an adequate research training package was not produced. We 
feel that the problem was primarily in the management procedures at BYU. 
A great deal was learned from this project and our subsequent experience. 
There is no question in our mind at this tlmfi that a team approach does , and 
can work; considerable revisions in our management structure and our own 
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team structure with tiie use of graducue students and the use of authors, 
have resulted from this project and others. We regret the very painful 
experience, but express our appreciation to AERA and the Office of Education 
for the opportunity to be involved in this learning experience. We hope that 
our production of products in other areas, other funded projects as well as 
those internally funded , .will demonstrate to the association and to the 
U.S. Office of Education our ultimate capability, and that this particular 
failure will not handicap relationships with these organizations in the future. 
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TABLE 1 



Team organization at BYU Dept. IRD Sept, 1972 



PROJECT COORDINATION 
M , David Merrill PhD 
Grant Barton PhD 
Reba Keele 
William Low 



EVALUATION 

Adrian Van Mondfrans PhD 
Richard Kay PhD 
David Butler 
William Quinn 



SYSTEMS DESIGN 

Harvey Black PhD 
Barbara Vance PhD 
Andrew Gibbons 
Roy Bennion 
William Endsley 
Don Martin 
Ireme Chuang 



IMPLEMENTATION 

Grant V. Harrison PhD 
Leo Garcia 
Norman Murray 
John Wilkinson 



PRODUCTION 

Edward Green PhD 
Junius Bennion 
C, Eric Ott 
David Tuttle 
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EACH PRIMARY DEVELOPMENT ACTIVITY IN FIGURE 1 
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ACTIVITY 


PRODUCT 


PERSONNEL 


MAN POWER - (MAN DAYS) ! 
TYPE ESTIMATED ACTUAL 1 


COSTS 


SHOES HOP PERK 

3-8 Content 
Analysis 


3D - MAY - SEPT 

No Formal Docu 

ment 
Produced . . . 

only outline & 

notes 


1972 

• Proj . director = 

Author = 

Inst. Ps ycholo- 
gist = 


Technical — 46.5 
Professional — 12 

Support — 29 

Author costs for 2 trips and 3 days 
are not Included. 

Subtotal 


$1400 
953 
470 

$2 ,823 


Project Coordi- 
nation 

7EAM & CRISIS 


No Documents 


Proj . Director = 

Dept. Manage- 
ment = 

U^.^^llk 


Technical — 47 

Professional — 10 

Subtotal 


1,410 

800 
$2,210 


4 • Systems 
Design 


"Needs & 
Goals" 
"Systems Deslg 
bpe c IX ica Lions 


Design Team; 

n 

. .... 


Professional 10 11.5 

Technical 20 5.5 

Support — 15 

Materials — $ 131 

Subtotal 

_ : 


920 
165 1 

240 

131 
$1,456 

! 


9 . Instructional 
Development 

9 

ERLC 


"Version I 
Mockup" 

"Version II 
Mockup" 

ft 


Development 
Team 


Professional 4 17.5 
Technical 34 9 
Support — 25 
Materials — $217 
Simulation Games 750 

TOTAL 


■■ 

1,400 

270 

416 

217 

750 

1 $3,053 
$9,542 



BACK TO SHOESHOP PERIOD FEB - JUNE 1973 









'Version III 


Development 


I 

1 

Professional — 


25 


^1 Revision 


Mockup 


Team 










Technlrfll 


1 u 








Support 


10 








Materials 


$526 



1,993 

281 
164 
526 
$2,962 



/ 
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MODEL D: DEVELOPMENT OF INSTRUCTIONAL PACKAGES - SECOND VARIANT 

W. Jarncs Popham 

As clu' project progressed, It became apparent that carelul :.L;i\ \ 
monitoring of the project's financial resources had created the possi- 
bility of exploring an alternative approach to the preparation of 
instructional materials, that is, an alternative to the BYU- supervised 
development of two trial-revised replicable instructional sequences. 
The projected cost of the two planned instructional products was approx- 
imately $12,000 each. It appeared that a sum of about $10,000 was available 
to support the creation of a second variation of instructional materials 
development . 

The Research Training Committee agreed to attempt to secure a series 
of low cost, high quality instructional booklets of approximately 75 
pages (typed, double spaced) length. Because the committee was anxious 
to focus these materials on a high need area, it was decided to emphasize 
the theme of educational evaluation and to prepare materials suitable for 
the training (and re- training) of the many individuals who engage in 
systematic evaluation at all levels of the educational enterprise. In 
particular, the committee wished to identify individuals who would prepare 
instructional booklets dealing with recent advances in evaluation techniques, 
strategies, and ideologies. Ideally, these authors would be leaders in 
the evaluation field who were both conversant with cutting edge thinking 
and also were able to translate those views into an instructional booklet 
comprehensible to evaluation practitioners. 
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The general scheme for securing the instructional booklets was 
as follows: 

1. The committee chairman would assume supervisory responsibility 
for the activity, 

2. The committee identified a number of potential authors and topics 
for consideration. 

3. The chairman contacted eleven individuals, inviting them to 
participate in the project. (One declined because of a sabbatical 
leave abroad.) 

4. Each author, upon agreeing to prepare the specified instructional 
booklet was to receive a $250 payment. 

5. Upon submission of the booklet manuscript, an additional $500 
pa3mient would be made. 

6. Upon acceptance of the manuscript (after review by the committee 
chairman and, if necessary, other committee members) a final $250 
payment would be made. 

7. Each manuscript would be reviewed by a subject matter expert, 
appointed by the Association's Publications Committee, before the 
AERA imprimatur could be placed on any subsequent publication. 

8. It was anticipated that the series of separate booklets, after 
approval by NIE authorities, would be released for distribution 
to a commercial publisher. 

The authors and titles of these instructional booklets are given below: 

Michael Scri ven, University of California , Berkeley- - Questions and 
Answers for Evaluators 
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Peter Airasian, Boston College- - Dcsi^nin^ Evaluation Studies 

Daniel Stufflebeam, Western Michigan Universi ty-- Al ternative 
Conceptions of Evaluation 

Gilbert Sax, University of Washingcon-- The Role of Standardized 
Tests in Evaluation 

Emil J. Haller, Cornell University^- Costs & Costing Procedures in 
Program Evaluation 

Richard Wolf, Teachers College, Columbia University-- Data Analysis 
and Presentation in Evaluation 

Jason Millman, Cornell University- - Criterion- Referenced Measures 

- Eva Baker, University of California, Los Angeles- - Formative 
Evaluation Techniques 

Kenneth Sirotnlk, University of California, Los Angeles-- Mul tiple 
Matrix Sampling 

Joel Weiss, Ontario Institute for Studies in Education- - De signing 
Affective Measures 

At the time when the project's final report was written, all manuscripts 
except those two noted by asterisks had been received and approved. Several 
of the manuscripts were reviewed In early draft stages. One manuscript 
was returned for basic modification. All manuscripts submitted were 
approved by the committee chairman. 
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i'VALUATfON OF MODKL A: TKAVKL I NG TRATNING INSTITUTES 
W. James l^opham 

The Concept 

Unlike the more familiar presession/postsession training model which the 
Association has conducted since the mid-sixties, the traveling training insti- 
tute was conceived of as a vehicle which would, in a geographical sense, bring 
training to members of the R&D community, not oblige them to seek it out once 
a year in conjunction with a professional association's annual m.eeting. By 
staging brief training institutes in various parts of the country, it was be- 
lieved that more R&D personnel could conveniently take advantage of relatively 
nearby opportunities to improve their professional skills - 

Further, since a given institute was tentatively scheduled for several 
offerings in different parts of the country and at various times, evaluations 
of the initial institute in each series would provide an opportunity for the 
supervisory agency (the Association in this instance) to exercise quality control 
over the offering of future sessions. In the most pleasant circumstances this 
would mean that the staff of a continuing institute could be provided with for- 
mative evaluation inputs which would result in improvements in future sessions. 
In a less happy case, this option would permit the cancellation of an institute 
so that an ineffectual session would not be repeated. 
The Institutes Offered 

In review, there were five traveling training institutes approved: (1) a two 
day session on evaluation directed by Michael Scriven and Daniel Stuff lebeam, (2) 
a four day session on Bayesian statistics directed by MelviJi Novick and Donald 
Meyer, (3) a three day session on performance-based training and assessment direc- 
ted by Frederick McDonald, (4) a three day session on written instruction direc- 
ted by Ernest Rothkopf and Lawrence Frase, and (5) a two day session on reading 



47 



directed by Jay Samuels. 

The first three of these sessions were conducted on one or more occasions. 
The latter two were not conducted due to insufficient advance participant enrcli- 
ments. A summary of the evaluations of the three sessions which were conducted 
will be provided in subsequent paragraphs, but first a brief examination of the 
evaluation procedures employed will be provided. 
Evaluating the Traveling Training Institutes 

There were three major purposes for evaluating the traveling training insti- 
tutes. First, in a summative sense, a judgment had to be made regarding whether 
to offer a given session subsequently on the basis of its earlier performance (s) . 
Second, more formatively, improvement-focused evaluation data were needed in or- 
der to make subsequent sessions of an institute more effective. Finally, an eval- 
uation of the traveling training institute concept itself had to be made. 

Complete evaluation reports for each of the individual sessions conducted 
are available in an appendix to this report. Since a complete description of 
the evaluation procedures and the evaluation results for each session are avail- 
able in these documents this section of the report will provide only highlights 
of these evaluations. 

For each session an evaluator designated by the staff supervising the project 
attended the institute, administered a number of measures, and prepared an eval- 
uation report, for that session. These on-site reports were subsequently submit- 
ted to the Association's Research Training Committee (for a go/no go decision on 
future sessions) and to the institute staffs (for improvement purposes). 

The Scriven-Sbuf f lebeam Institute. This session was offered in Portland 
(October, 1972) for 23 participajits , in Tampa (Jajiuary, 1973) for 26 participants, 
^ in Tucson (March, 1973) for 19 participants, and in Berkeley (July, 1973) for 

f 

17 participants. Originally scheduled for only three sites, the positive re- 
Q sponse of previous partipants resulted in the offering of the fourth session in 

ERIC 

Mirfiiirf i mi i in Berkeley. 
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'^hc relatively small number of participants in each session stemmed chiefly 
I 

from the directors imposed ceilings on attendance. In particular, for the first 
two sessions in the series almost twice as many individuals applied as were ad- 
mitted. The combination of the directors' prominence plus the current interest 
in the topic of evaluation undoubtedly contributed to the high participant in- 
terest in the session. 

In general, the sessions were well received by participants, with an over- 
whelming number of those attending providing anonymous responses that they would 
be willing to attend the session again and to recommend the session to a colleague. 
The third session in the series (Tucson) was less successful then the other three, 

apparently due to Scriven»s day-late arrival and Stuff lebeam* s somewhat early 
departure, A number of suggestions were offered by the on-site evaluators for 
modification of the session, and the majority of these were attended to by the 
institute staff. 

The Novick-Meyer Institute. Tnis session was offered in Amherst, Massachusetts 
(October, 1972) for 16 participants and in Tampa (January, 1973) for 26 participants. 
A third session scheduled for Eugene, Oregon was cancelled, due to insufficient 
pre-registration (only five applicants) . Undoubtedly the more limited appeal of 
this institute, coupled with the somewhat difficult access of its third site, led 
to the insufficient number of participants for the intended third session. 

The first two sessions were well received, with almost all of the participants 
completing anonymous questionnaires reflecting a willingness to attend the session 
again and to recommend it to a colleague. Use of a cognitive test indicated con- 
siderable pretest to posttest increases in participants' knowledge of the insti- 
tute's content. A number of the suggestions emanating from the first session's 
evaluation were followed in the second session. Although, perhaps because of the 
integral role played by computer equipment in this institute, resolution of logistical 
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and budgetary considerations for this session proved difficult, the two institutes 
appeared to be quite successful. 

The McDOiiald Institute . This session was offered for three days in Boston 
(May, 1973) for 25 participants. Althrough it had been anticipated that the 
institute would be offered on several subsequent occasions, the intense partici- 
pant dissatisfaction with the quality of the session led the Association's Research 
Training Conunittee, after an examination of the evaluation data from the first ses- 
sion, to reach a unanimous decision to cancel any future offerings of the institute. 

Only three of the 23 participants who completed the anonymous end-of -insti tute 
questionnaire indicated that (if they were able to choose again) they would attend 
the institute. Only four indicated they would recommend the institute to a 
colleague. Several letters registering extreme unhappiness with the quality of 
the session were received by the Association. It was generally contended by 
participants that there was insufficient preparation by the staff and inadequate 
content to be presented. It was apparently most frustrating to pay almost $50 
per day, only to be placed in small group sessions in order to share problems 
with other $50-per-day participants. Approximately 75% of the structured class 
time was spent in closureless small group sessions. One of the scheduled in- 
structors failed to attend the session because of a scheduling confusion. The 
director was obliged to be absent for one day. These and other factors operated 
to make this institute, at least in the view of the monitoring Research Training 
Committee, a rather dismal failiire. Fortunately, the staggered nature of the 
traveling trainincf institutes permitted the luxury of cancelling future sessions 
of the institute. 

The Samuels Institute . This two day session focused on reading acquisition 
with an emphasis on deprived populations. Two sessions were scheduled for summer, 
1973, one in New York during mid-June and a second in Minnesota during late July. 
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Only fiv^' af^il ications were receivod for the New York session and only four 
ap[)licationh> for tho Minnesota session. Neither session was approved due to 
insufficient interest. 

The Rothkopf-Frase Institute. This three day session dealing with research 
in written instruction was scheduled for Madison, Wisconsin during August, 1973. 
Of the 12 applications received, the directors viewed only six to reflect ad- 
equate entry skills, hence this session too was cancelled due to inadequate pre- 
registration . 
Evaluating the Concept 

One factor to be considered in evaluating the merits of the traveling training 
institute as a vehicle for "bringing training to those needing it" is the type of 
people the sessions served. Although a better picture can be gained by consulting 
the individual evaluation reports, it appears that the parLicipants were varied 
but generally resenibled the kind of individuals who attend the presessions and 
postsessions conducted in conjunction with the annual meeting. 

Most interesting to the various individuals serving as e valuators was the 
considerable distance traveled by participants to attend the institutes. In all 
of the Scriven-Stuf f lebeam sessions and one of the Novick-Meyer sessions 50 per 
Cf:nt or more of the participants traveled 1,000 or more miles to attend the 
institutes. If the sessions were conceived of as "regional in nature", then one 
must employ a liberal definition of what constitutes a geographic region. 

Another point of interest is that although participants expended considerable 
funds on the sessions, few if any spent any substantial amount of personal funds 
for the institutes. Sponsoring agencies, almost without exception, paid partici- 
pamts * expenses. 
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Participants' Appraisal of '::he Traveling Training Institute Concept * 

A critique form was administered to participants at the conclusion 

of each of the first seven traveling institutes. This instrument 

included a question regarding the advantages and disadvantages of the 

traveling institute concept as compared with those of the AERA annual 

meeting pre- and postsessions . 

This section is a summary of the responses to this question collected 

at five traveling institutes."^ Several participants responded with 

reference to characteristics specific to the institute they attended 

(e. q, , daily scheduling was poor); these concerns have been included 

in the evaluation reports for each institute, and will be considered 

inappropriate for this summary. Of the 101 available instruments, 68 

included responses citing advantages and 30 listing disadvantages 

(excluding inappropriate responses) . Responses given by three or more 

participants are listed in Table 1. 

The most prevalent (17) listed advantage of the traveling institute 

format was that it requires fewer days away from work at one time. It 

was generally expressed that it is more convenient to take two short 

absences from work in order to attend the convention and a training 

session separately than to be absent for one long period in order to 

^•This section was prepared by Marilyn Averill of the University 
of Colorado, Laboratory of Educational Research. 

■^Information was available from the following institutes: Alternative 
Conceptions of Evaluation given in Portland, Oregon; Tucson, Arizona; 
and Berkeley, California; Bayesian Statistics held in Tampa, Florida; 
and Performance Based Teacher Education given in Boston, Massachusetts. 
Responses have unfortunately been misplaced from the Evaluation Institute 
in Tampa, Florida, and the Bayesian Institute in Amherst, Massachusetts. 
However, a review of the data from the remaining five institutes shows 
no systematic differences in responses across institute titles or locations. 



attend them together. 

Reduced costs were listed as an advantage by eleven participants; 
most referred specifically to reduced transportation costs resulting 
from the regional locations. Choice and convenience of locations and 
dates were seen ar, an advantage by four participants. Ten others listed 
convenience of location alone, without reference to cost or dates. 
Seven participants cited choice of dates as an advantage; in addition, 
three others stated specifically that the choice of dates made training 
sessions more responsive to immediate professional needs. 

Fourteen participants indicated that benefits were obtained by 
holding training sessions separately from the convention because of 
distractions created by other activities at the convention (7) , and 
because of the "information overload" and generally tiring effects of 
convention attendance (7) . 

The only disadvantage listed with any consistency (22) was that 
of the duplication of travel expenses incurred from attendance at 
both a traveling institute and the annual meeting. Apparently the 
reduced costs listed as an advantage occurred only for those people 
who did not also attend the AERA convention. Other disadvantages 
cited tended to refer to specific institutes. 



Table 1 

Participants responses concerning the advantages and disadvantages 
of the traveling institutes as compared with pre- and postsessions 
associated with the annual convention. 



Advantages Frequency 

1. ) Less time away from work 17 

2. ) Convenient locations 10 

3. ) Reduced costs 11 

4. ) Choice of dates 7 

5. ) Fewer distractions 7 

6. ) Convention too exhausting 7 

7. ) Choice and convenience of dates and location 4 

8. ) Responsive to immediate professional needs 3 



Disadvantages 

1,) Extra travel expenses 
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Survey Data 

Approximately 74 percent of the respondents to the survey of AERA 
members were aware of the traveling institutes, although only 3.7 percent 
attended one of the sessions. The Educational Researcher was the primary 
m.eans by which most respondents learned of the existence of the traveling 
institutes . 

Major reasons cited by respondents for not attending any traveling 
institutes include inability to take time off from work (33%), distance to 
institute (24%) , the cost of the institute (21%) , and the inappropriat eness 
of the topics for the participant (19%) . If one assumes that concern with 
distance is in many cases a concern over travel costs, it is apparent that 
the overall costs for the institutes are uncomfortably high. This is 
reinforced by the fact that only one percent of the respondents thought a 
r<'G of $50 or more a day was appropriate, while the remaining 99 percent 
\ t a lesser fee would be appropriate, with the large majorit:y choosing a 
lee of $20 to $30 a day. 

Support for offering training sessions at times other than the annual 
meeting exists in participant data which shows that more participants would 
be available to attend a summer training session (51 percent of the respondents) 
than at any other time of year. Next choices are winter (34%), spring (22%), 
and fall (21%) 



EVALUATION OF MODEL B: ANNUAL MEETING PRESESSIONS 
W, James Popham 

The Concept 

Although there is a long and rather successful history of training sessions 
offered in conjunction with the annual AERA meeting, there has always been some 
concern that such sessions are reaching only the same sorts of people or, worse, 
that the sessions may be reaching only an identical group of "repeater" partic- 
ipants. The training model to be tested here involved the preparation of annual 
meeting training sessions which were specifically designed for audiences other 
than those individuals who typically attend an AERA meeting. 

In essence, then, this particular training model attempted to incorporate 
a previously employed format, namely, presessions associated T/ith the annual 
meeting, but directed that format toward a unique audience, that is, members of th( 
R&D community who would not characteristically attend a traditional AERA pre- 
session or postsession. 
The Training Ses'^ions Offered 

From a number of proposals whic]i were submitted to the AERA Research 
Training Committee, two were selected in order to implement Model B, The 
first of these, entitled "Research on Reading Acquisition: With An Emphasis 
On Deprived Populations," was a session directed by Professor S. J. Samuels 
of the University of Minnesota. This was a five day presession designed for 
a wide array of R&D specialists concerned with reading, particularly those 
working with deprived populations. According to the announcement of the 
session appearing in the Educational Researcher , the directors of the session 
considered it "suitable for curriculum supervisors, teachers of learning-disabled 
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studentLS, personnel who work with inner-city residents and the disadvantaged, 
school evaluators, curriculum designers, reading specialists, and researchers 
in verbal learning/' In addition to Professor Samuels, four well known instruc- 
tors constituted the staff for this session. 

The second Model B session was a three day presession entitled "Coinputer 
Managed Instruction." This session was directed by Dr. Frank B. Baker of 
the University of Wisconsin who was assisted by three well established collegues 
as instrT^ctors , The primary "new" audience for the second session was the 
public school educator who needed to acquire a working knowledge of computer 
managed instruction, both in order to participate in research related to ^his 
endeavor as well as designing and conducting instructional systems using a computer 
management base. 

Unfortunately, only one of the two sessions offered was actually 
conducted. The Samuels session attracted 14 participants. That number, because 
of the substantial participant tuition charge for a five day session, provided 
sufficient tuition to make the session essentially self-supporting. The Baker 
session, on the other hand, received only seven applicants and was not offered. 
Evaluating the-.Samuels Presession 

A separate evaluation of the Samuels presession is enclosed in the appendix. 
The interested reader is advised to consult that more detailed appraisal. The 
highlights, however, of the evaluation can be described here briefly. 

The most important question regarding this presession was whether the 
session reached the clientele for which it was intended. An examination of the 
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14 participants at the session reveals that six were associated with a college 
or university, three were from public school systems, one was from the State 
Department of Education, two were students, and two represented other 
categories. It appears that, on the basis of a very small number of participants, 
the session was only moderately sucessful in reaching people other than the more 
traditional participants attending the annual meeting. 

Regarding the participant's reaction to the session, there appears to be 
a reasonable level of satisfaction but by no means enthusiastic approval of 
the session. When thp- participants were asked whether, if they were able, they 
would choose again to attend the session, five indicated yes, two indicated no, 
and five were not certain. (Two of the 14 participants did not complete the 
final evaluation form.) When asked if they would recommend this session to 
a collegue if it were offered again, eight indicated yes, threfi indicated no, 
and one was uncertain. As -f licated above, detailed analysis of participant 
responses of the session is included in the separate appendix, but responses 
to their general types of evaluative questions appear to reflect a generally 
positive reaction to the session. 
Evaluating The Concept 

One thing appears to be clear from the two trails of Model B, namely, 
that merely offering sessions which appear to be suitable for targeted R&D 
audiences does not automatically produce a large attendance. The Research 
Training Committee in screening the proposals which were designed to implement 
Model B relied somewhat passively upon proposals submitted to it. Rather than 

9 
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actively soliciting proposals from individuals who could design targeted 
training sessions, the conunittee awaited the receipt of such proposals. 
There were not many eligible proposals that were submitted. Apparently, the 
two sessions selected were not, on the basis of their content and staff 
alone, sufficiently appealing to attract many of the individuals for whom 
they were designed. The fact that one of the two sessions offered was not 
even conducted and that the other drew only 14 participants suggests that 
the Research Training Committee did not select sessions with sufficient 
audience appeal* Apparently, a .nore aggressive committee stance is required 
to locate training sessions which will be more attractive to groups other 
than the individual R&D specialists who attend the annual AERA meeting. 

In addition, it appears that publicizing such sessions in our 
customary channels (e.g., announcements in the Educational Researcher ) is 
inadequate. Of course, this consideration is confounded with the nature 
of the sessions themselves, but the low attendance would also suggest that 
more intensive promotional campaigns have to be mounted in association with 
the new training ventures of this sort. 

Model B, as it was conducted in connection with this training program, 
appears to be a potentially useful scheme, but warrants further, more carefu 
testing. 
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EVALUATION OF MODEL C: ANNUAL MEETING 
TRAINING ACTIVITIES 
Blaine Worthen 

The Concept 

Although AERA has established a familiar pattern of offering training 
shortly before or after annual meetings through the familiar presession/ 
postsession training model, there has been little or no conscious effort 
prior to 1973 to provide direct training *"o Association members and otlier 
professionals at AERA annual meetings. In 1973, the Association supplemented 
the usual reports of research and evaluation studies and discussions of 
evolving methods and techniques with a series of activities designed ex- 
pressly to provide explicit training during the annual meeting. 

It was believed that training opportunities during the annual meeting 
were especially desirable because large numbers of potential trainees and 
trainers would already be scheduled to attend annual meetings, eliminating 
problems associated with scheduling alternative times when it might be more 
difficult or costly to bring busy trainers and trainees together. Conversely, 
the existence of many competing activities during annual meetings was 
viewed as a potential drawback to training during the annual meeting. There- 
fore, it was determined that Model C should consii.t of training activities 
which were relatively short in duration, readily accessible to potential 
trainees, and scheduled on either a repeating basis or at times thought to 
conflict minimally with competing alternatives. 
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The purpose of Model C and its evaluation was to determine whether 
explicit training could be successfully conducted during the annual 
meeting, both as a test of (a) the specific training activities conclufted 
and (b) the concept of annual meeting training. 
An nual Meeting Training Activities 

Model C comprised three separate training activities. The first was a 
series of three mini-courses (the fourth was cancelled due to the sudden 
illness of the director). Each session was on a specific topic in educa- 
tional research and was three to four hour? in duration. (1) Eva L. Baker, 
Planning Formative Evaluations for Instructional Products; (2) Benjamin 
Wright, Sample-Free Item Calibration £i Test Free Person Measurement; and 
(3) Ronald G. Ragsdale & Saul Reisman, Computers in Educational Research. 

The second activity was a series of four conversation hours, each of 
which provided for an hour or two of informal conversations between 
invited speakers and any participants interested in discussing topics of 
their choosing with the speakers. Invited speakers were Robert M. Gagne, 
Robert L. Ebel, Richard C. Anderson, and Roger Heyns. 

The third activity was provision of training materials in a combination 
training exhibit and study center. The intent was to identify and collect 
quality research training materials which participants could study either to 
learn concepts presented in the materials or to determine whether the materials 
might be useful in training activities they planned to conduct in the future. 

A more complete rationale for each of these Model C activities is 
presented earlier in this report along with a description of procedures used 
in soliciting and selecting trainers and products and publicizing the activities. 



Evaluating Annual Meeting Training Activities 

There were three major purposes for evaluating the training activities 
conducted during the annaal meeting. First, each single activity (e.g., 
Baker's mini-course) was evaluated to determine whether to offer it or 
something like it again at future annual meetings. Second, each set of 
activities (e.g., conversation hours, viewed as a whole) was evaluated for 
the same purpose. Third, an evaluation of the annual meeting training 
concept itself was necessary. 

A complete detailed evaluation report of Model C-is available in the 
Appendix. Therefore, only a brief summary of evaluation procedures and 
resulta will be provided here and in subsequent sections. 

The evaluation plan for Model C differed somewhat across the three 
activities. Data were collected in both the mini-courses and conversation 
hours by three techniques: (1) eliciting participant background information 
and reactions using short questionnaires and rating scales; (2) eliciting 
staff reactions, using a short rating scale; and (3) direct observation by 
a graduate student who served as a participant-observer in each session. 
Because of the short duration of the activities, data collected by the first 
two techniques were not as extensive as might have been possible given more 
time. Therefore, the use of observers was designed to supplement the other 
data and note irregularities or problems in the sessions not covered by 
the other instruments . 
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For the training exhibit/study center, the plan was to register the 
name and address of each person who entered the exhibit area and send a 
questionnaire to him later to elicit reactions to the materials and benefils 
received from perusing them. 

An ancillary survey of 600 AERA members was conducted to determine 
whether members who did not attend any of the training activities were 
aware of their existence and, if so, why they chose not to attend. A more 
detailed description of that survey is contained in the Appendix. The results 
of implementing these procedures are discussed below, with each of the 
three training variations in Model C treated separately prior to discussing 
the utility of the overall concept of annual meeting training. 
Mini-courses 

The three mini-courses were relatively well attended, with the following 
number of participants completing the instruments at the end of each session: 
Wright (16) , Baker (38) and Ragsdale (17) . Numerous other persons were 
present for part of the sessions, but (for reasons described later) left 
before the sessions were completed. An analysis of the background of 
participants showed that twice as many held membership in Division D as in 
any other division, suggesting a possible selection bias in the topics for 
the mini-courses. Most participants attended cut of a desire to gain new 
skills or improve existing skills, rather than to discuss their individual 
research or evaluation problems with experts. A majority of the participants 
indicated they had limited or no experience in the respective topics. It 
is significant that 65 percent of the participants first learned about the 



iDini-courses thrcugh the Annual Meeting Program and another 14 percent 
first learn^jd of those sessions through flyers handed out during the 
convention. Only 20 percent learned of the sessions prior to the 
annual meeting and only 9 percent of these were made aware by reading an 
announcement in the Educational Researcher. 

The survey of AERA members at large showed that approximately eight 
percent of those persons who attended the annual meeting attended a mini- 
course, 20 percent attended a conversation hour, and 16 percent visited 
the trair:ing exhibit. Approximately 38 percent indicated they were too 
busy to attend a mini-course, conversation hour, or training exhibit. 

Numerous logistical problems plagued the actual operation of the mini- 
courses; these centered on problems with facilities and equipment and can 
be attributed to inadequate coordination of hotel and annual meeting 
staff. The room for the Baker session was far too small for the 75 to 80 
persons who were in attendance shortly after that session began. As a 
consequence, physical discomfort and inability to see or hear adequately 
led to about half of the participants leaving prior to the end of the 
session. Similar problems occurred, but to a much lesser extent, in the 
Wright and Ragsdale sessions. The Ragsdale session was disturbed by latei 
delivery of visual aid equipment, as well as a last minute change of locale 
to another room in another hotel. Mini-course staff indicated that these 
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difficulties, combined with tight time allotments, forced them to 
significantly alter their presentation of the content of their courses. 

Despite these difficulties, the mini-courses were for the most pare 
enthusiastically received by the participants who remained to the end of 
the sessions. In general, the topics of the mini-courses were viewed by 
participants as important and the courses themselves as very useful in 
their learning about the topics. Participants liked the convenience (^f 
scheduling the mini-courses during the annual meeting, although a majority 
felt that the courses should be lengthened somewhat. 

Overall, the participants indicated that the instructors were very 
well qualified, their presentations well prepared, and the instruction 
interesting and informative. Participants reacted very negatively to 
the facilities in which the mini-courses were offered. On the average, 
participants felt the tution fee was reasonable. 
Conversation Hours 

The conversation hours were well attended, with over 100 persons in 
attendance at one time in the Gagne session (in a room designed to seat 
40). Because persons came and went at will, the evaluation instrument 
administered at the end of tl;o session was completed by only a sub-set 
of the persons who attended, which may have introduced some bias into the 
data. The number of participants who completed instruments at the end of 
each session were as follows: Ebel (34), Anderson (23), Gagne (23), Heyn.= 
(17). Most of these respondents were in attendance for an hour or more of 
the conversation , 
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The conversation hours were designed to provide opportunities for 
practicing researchers to interact with luminaries in their fields in much 
the same way AERA graduate student seminars provided comparable opportun i t. ies 
for graduate students. Eighty-four percent of the participants in conversation 
hours were employed (16 percent were full-time students) and 61 percent held 
the doctorate. A majority said they had considerable or a great deal of 
experience in the field of the discussion leader. Divisions C and U were 
most heavily represented, with over half of the participants holding member- 
ship in those divisions. This slightly overrepresents the divisional member- 
ship of AERA which contain approximately 45 percent of the members within 
those two divisions. 

Again, most of the participants first learned about the conversation 
hours through the annual meeting program (69 percent) or at the annual 
meeting (16 percent). The idea of having participants send questions to 
featured speakers before the annual meeting must be judged a failure, since 
only four percent of the participants contacted the featured speakers by 
mail pr ior to the conversation^ 'ht)ur . The concept of having "big name" 
speakers was obviously successful, since three out of four participants at- 
tended primarily to hear the featured speaker. 

The content of the conversation hours was judged by participants tc; ho. 
important, the discussion leaders very well qualified and receptive to other 
points of view, and the discussions informative and interesting. There was a 
strong sentiment among participants that conversation hours should be offered 

at future annual meetings. 
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Training Exhibi t/ Study Center 

There are some indications that the training exhibit concept has 
potential, i_I. ^ sufficient quantity of quality training materials can be 
identified to warrant its continuation. For example, approximately 100 
persons perused at least some of the training products every day. Their 
verbal comments to AERA staff were primarily positive, with the exception 
that many persons felt a separate room should be set up for films, which 
otherwise distract persons working on printed materials or listening to 
cassette tapes. 

Unfortunately, no further evaluative data are available because of a 
comedy (or tragedy) of errors associated with the evaluation of the training 
exhibit. The list of names and addresses of persons who visited the exhibit 
(and who would therefore have received a questionnaire) inadvertently was 
mixed in with an exhibitor's lists of orders for his materials and was carried 
off and subsequently lost by the exhibitor. Observer notes which might 
have helped to fill the gap disappeared when an erroneously scheduled Division 
C meeting was held in the exhibit area and a majority of the training pro- 
ducts (and even the observer notesi) departed with the attendees. All of 
which is a sad commentary'' on both the acquisitive tendencies of some of our 
colleagues and the resultant need to develop new theft-proof evaluation 
techniques. In the interim, there is no good way to judge the success of 
this annual meeting training effort. 

The Concept of Annual Meeting Training 

Based on the evaluative data available, the concept of providing training 
at the annual meeting must be judged at least a qualified success. The attempts 
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to provide training at the annual meeting were well received by participants. 
Spe'cif ically , the mini-courses were rated very positively by participants, and 
it appears that a wider offering of such courses might be in order for the 
next annual meeting. Although the numbers of persons in attendance at 
conversation hours changed the intended nature of those sessions and may 
have diluted the amount of training received by any one participant ^ the 
reactions to the sessions were very positive and participants clearly stated 
their hope that such sessions could be continued. It would seem worthwhile 
for AERA to offer expanded training opportunities at the next annual meeting. 

Some recommendations seem in order. First, better publicity should 
be given to mini-courses and conversation hours in the annual program, since 
that is the primary vehicle through which participants became aware of those 
sessions. Better control of logistics (scheduling and security of rooms, 
etc.) for special annual meeting training events is a must. Mini-courses 
of somewhat longer duration should be offered to assess their utility, in 
addition to continuing with some of the present length. Each Division 
might be invited to work cooperatively with the Research Training Committee 
to sponsor one mini-cou; se and one conversation hour. The selection of 
eminent persons to serve as featured speakers in conversation hours should 
be continued. 



68 



EVALUATION OF MODEL D: DEVELOPMENT OF INSTRUCTIONAL PACKAGES 

W. James Popham 

There were two separate approaches to exploring the efficacy ol' 
Model D< The first of these was the effort to develop two replicable 
products at the Brigham Young University Division of Insiiructional 
Research, Development, and Evaluation. Earlier in this report the director 
of that project provided an insightful analysis of the difficulties 
associated with the endeavor. From the point of view of the Research 
Training Committee, the group charged with monitoring the quality of the 
under-development projects, this phase of t±ie project proved particularly 
vexing. There was a long history of unsatisfied expectations, overdue 
progress reports, and low quality products. Finally, after deciding that 
the first of the two products was unacceptable but potentially a learning 
experience for the BYU group, the committee demanded highly explicit 
specifications for the second product. When these specifications were, in 
several successive versions, judged inadequate with respect to both specificity 
as well as content significance, a telephone and mail poll of the committee 
resulted in a decision to terminate all work on the second product. The 
committee preferred to return the unexpended funds to the National Institute 
of Education rather than experience a second result similar to that associated 
with the initial product. 

It is likely, of course, that the unfortunate result of this effort 
stems chiefly from the ideosyncratic situation at BYU so well described 
earlier. Yet, another interpretation is possible. It may be that it 
is next to impossible to locate a development group that is both (a) capable 
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of producing a high quality instructional product at modest cost and (b) 
available to assume the task. 

In other words, the effective instructional development laboratories 
will typically be too heavily committed to take on the one or two short 
types of instructional development tasks originally contemplated in Model 

The second variant of Model D, the production of low cost instructional 
booklets, was initiated so late in the project's existence that it is 
difficult to evaluate them properly. From a raw cost/product standpoint, 
the scheme appears potentially effective. A large number of instructional 
booklets on current topics was prepared by many of the nation's leading 
evaluation experts. In part, one suspects that not only the prospect of 
a $1,000 honorarium but also the opportunity to prepare an instructional 
document for their professional association were key factors in securing 
such a prestigious array of authors. 

In the chairman's view, the quality of all the manuscripts is suitable 
for publication, and several of them appear to be truly exceptional 
contributions to the field. Yet, the critical test of this form of develop- 
ment will depend upon the impact of the materials upon those individuals for 
whom they were designed. Perhaps the lack of a trial-revision development 
pattern will render the booklets ineffective. We shall temporarily have 
to defer our assessment of merit on the second variant of Model D. 



SUMMAHY 



In review, the entire project represented an effort to explore 
alternative vehicles for providing continuing education for members 
of a professional research association. The project was initiated 
in recognition of the clear need for providing training which can 
effectively combat obsolescence on the part of a professional 
research community. Several questions were under scrntiny in the 
investigation. First, ^,here was a focus on four specific training 
models which were employed. Second, there was attention given to 
the general question of whether the overall structural scheme for 
this training endeavor was viable, namely, whether a group of research 
professionals in the field (as represented by the AERA Research 
Training Committee) could work in collaboration with the Association's 
central office staff in order to conduct reasonably effective training 
ventures. The final section of the report will deal with both of thes 
questions . 

With respect to each of the four training models, a more detailed 
analysis can, of course, be secured by consulting the sections of 
the report which describe those models and which evaluate them. In 
general, however, it appears that several of the models hold 
considerable promise, while one proved rather unsuccessful. Model A, 
tae traveling training institute was initiated during the period of 
the project. At least two were received and well attended. The 



tr/iveling training institute cc^ncept was organizt^d cm a self-support 

basis, thereby permitting the conduct of only those sessions which 

could, in essence, pay their ovn way. While this is a limitation, of 

course, it certainly suggests that the traveling training institute 

is a practicable scheme for reaching educational research professionals. 

It appeared that, contrary to the expectations of the Research Training 

Committee, the most important factor in participants' decision to 

attend the traveling training institutes was not the geographic location 

but, rather, the temporal appropriateness of the sessions. It is apparently 

easier for people to take several days away from their work for a 

traveling institute and then take other time off for attendance at 

the annual meeting, than to attend an annual meeting plus a pre- 

or postsession which results in a much longer period away from one's 

work. 

With respect to Model B, annual meeting presessions designed for 
broader audiences, the evidence is less encouraging. Two sessions 
were offered during the 1973 annual meeting, and only one of these 
received sufficient applications to actually conduct the session. At 
that, only 14 participants attended the particular presession. It appears, 
on the basis of this modest trial > that It might make more sense, if 
one is attempting to reach divergent research audiences (such as those 
individuals whose primary professional affiliation is the International 
Reading Association rather than AERA) , to actually conduct sessions 
prior to or following the meeting of other professional groups (such as 
the International Reading Association) . It is apparently unlikely that 



the availability of a special presession or postsession in conjunction 
with the AERA annual meeting was sufficiently appealing to get individu«'il 
to attend the AERA Annual Meeting. 

The special training activities arranged during the annual meeting, 
Model C, seemed (along with Model A) to be the most promising of those 
training schemes considered in this project. The response of members to 
the during-meeting mini training sessions and conversational hours was 
particularly gratifying. As a consequence of this project, the AERA 
Research Training Committee plans to expand markedly the number of 
training activities offered in conjunction with the annual meeting. As 
more experience is gained regarding the conduct of these sessions, one 
suspects that they will become an integral part of the annual meeting 
activities . 

Model D, the model focusing on the preparation of instructional 
materials, offered the least promising results. As was documented 
earlier in the report, efforts to produce two replicable instructional 
products for use by educational researcn practitioners met with 
considerable disappointment. Indeed, one of these two projects had 
to be cancelled prior to its completion. As was suggested in the 
section involving the evaluation of Model D, it is likely that highly 
qualified instructional development agencies (such as mature regional 
laboratories) are sufficiently committed to their own development 
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responsibilities that they cannot take on development projects such 
as those which a professional association might sporadically wish to 
undertake. Such mature development agencies would be reluctant to 
assign their mo st qualified personnel to such an intermittent production 
effort. 

At the same time, those groups which would be available for the 
development of instructional materials such as those undertaken in this 
project are, very likely, net the most effective agencies of this sort. 
As a consequence, the quality of the resulting instructional products 
may parallel those which were witnessed in this project. 

One suspects that an alternative vehicle for producing instructional 
materials which were both (1) trial-revision based (and therefore effective) 
and (2) of real utility to research practitioners, would be to establish 
a programmatic effort to develop instructional materials of this sort. In 
other words, long term funding would need to be provided, either for a 
new , modest-scope agency, or for some already established instructional 
development group. If a regional laboratory, e.g., foresaw the possibility 
of a three-five year project involving the preparation of a series of 
Instructional materials for educational research pactitioners , then 
appropriately talented staff could be assembled for the endeavor. Unless 
a scheme such as this is devised, ad hoc instructional development of this 
kind seen in the present project is unlikely to be successful. 



With respect to the second variation on Model D, the production 
of low-cost instructional booklets dealing with high import topics, 
the jury is still out. Clearly, through the prestige of the research 
association, one is able to secure highly talented members of the 
profession to prepare instructional booklets of this sort. Whether 
those same individuals would be willing to participate at greater 
length (and at greater cost) in the preparation of trial- revision 
materials remain to be seen. The kind of writing task which was 
requested of these authors was more '"-onsonant with their characteristic 
writing efforts and, therefore, was readily completed. When the 
instructional booklets have been distributed and reactions from the 
field have been secured, then we will be in a better position to say 
whether this particular variation of Model D is an effective one. 
At the moment, the evidence is not at hand. 

Regarding the general structure of the training project, a 
positive appraisal can be rendered. The organizational structure was 
definitely a viable one. The major policy decisions were made by members 
of the profession through their Research Training Committee and the 
administration was primarily carried out by the central office staff 
of that organization. One suspects, however, that the effectiveness 
of this particular relationship was more a function of the individuals 
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involved than of a generalizable organizational "discovery." The 
Research Training Qommittee and the central office staff worked v/ell 
together* Frequent letters and telephone calls made it possible to 
monitor the progress of the project, with a few minor exceptions, quite 
satisfactorily. As the need developed for additional scrutiny on 
certain points, we were able to secure the imput of other members of 
the profession.* One had the feeling that, had key members of either 
the AERA central office staff or the Research Training Cotnmittee been 
replaced by less conscientious individuals, the organizational structure 
effectiveness might have suffered dramatically. 

In retrospect, several of the training variants and organizational 
schemes devised as a consequence of this project seemed to offer 
considerable promise, not only for the American Educational 
Research Association, but for comparable professional research groups 
in other fields. There are clear lessons to be learned from this project 
and, quite obviously , there is a need for additional inquiry regarding 
not only certain of the models studied here (such as those which appear 
promising) but other training vehicles as well. In view of the fact 
that the focus of the educational researchers ' s activity is the improvement 
of education for millions of our nation's citizens, the stakes are too 
high to permit this kind of inquiry to be a one-shot case study. 



For example. Dr. Robert Gagne was persuaded to prepare a position paper 
dealing with an important aspect of our deliberations, namely, the 
degree to which competency assessment of research professionals would 
relate to the conduct of training activities such as those we vere 

supervising . 
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APPENDIX A 




Kv,i I ua t i on cif AEll(\ Tr;i..ling Training Institute 



ERIC 



"Bayesian Statistics and Interactive 
Computing Systems'^ 



Directed by 



Melvin R, Novick 
American College Testing Program 



and 



Donald Meyer 
University of Pittsburgh 



Oct. 6-9, 1972 
University of Massachusetts 



/ 



The dc'SL^n, dcvolopmcnf of Lns i irrnnts, and supervision of Lhc 
fV^iluation of the travclinj^ trainin i iir t. iLaiLos arc Lho rcsponsihil- 
ity of lUainc Worthcn, with the as:. - {.ancc^ of Marilyn Avcrill, a 
doctoral student at the Lab of Educ, . ( i.or a L RescMrcli, University of 
Colorado. I as t rumen t Number I roll-.' : ted dcmojr,r aphic i'lf oriRation 
from the participants. The second i isl rumen t was designed to assess 
cognitive outcomes of the Institute. Affective changes resulting 
from participant attendance at the I'lStitute was assessed by the 
third instrument. General evaluatio-^- data from both the staff and 
participants were gathered from two :ritiquc forms (staff instrument 
number I and participant ipstmment number IV). Instruments for a 
follow-up study of participants, non • attendees and non-applicants 
v;ill be administered to the appropri te groLips in approximately 
four mon ths . 

Instrument Administration and Analyses 

Instrument #1 was administered to each participant at the beginning 
of the Institute. At the same time, one iialf of tlie attendees were 
randomly selected to respond to Participant Instruments #2 and #3 
(the cognitive and affective measures); these instruments were also 
given to the entire group as a post-test at the conclusion of the 
Institute. Th resulting evaluation design appears below. 

R 0^ X 0^ 

R 

Participant Instrument #4 and Staff Instrument #1 were also admin- 
istered at the conclusion of the Institute to all persons in their 
respective groups. 

The following analyses were performed on the data: 

(1) descriptive statistics, comprising simple tabulations 
jith means and standard deviations where appropriate, 
for all instruments. 

(2) one way ANOVAs between total pre and post-test scores on 
Participant Instruments #2 and #3, for the following com- 
parisons: ^1 " ^2 > Ol 5 and O2 - 0^. 

(3) contingency tables; employing institution and total 
amount of money expended for the Institute by each parti- 
cipant were used as stratifying variables and plotted 
against responses to items 4, 23 and 24 on Participant 
Instrument #4. 



Due to the invited guest in attendance, these instruments were erron- 
eously administered to a few individuals. All but 2 of their responses 
were identified and eliminated from the final tabulations reporced. 
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1 ns I i (. n t <• P/] r t ic i n- in t s 



Pa r I i L i pin t s were- SL'K'Cl'd [or W u-ssi(M^ by iht- '.o-d i ric t^M- of 
the Ins t L ( iitr , Molvin Novick . Tin < i > 1 1 i'- a t inn form (rc^py *'nclosicl in 
iht' appendix) was the princi])al voUirl. .^nrpioyed to screen applicants. 
There were sixteen registered particinants attendipf^ ! he session. In 
addition, a limited number (approxim.! ; e J v eightO sindL-nts and faculty 
of the University of Massachusetts we rt. invited (on a complimentary 
has is) to attend the Institute. There was a considerable variation in 
the amount of time these individuals spent at the Institute during the 
four days . 

Table 1 contains .i description of p.irticij^ant s en several charac- 
teristics. The majority of participants were employed in institutions 
of h igher educa t ion , held advance def;rees , primarily eng"'ged in teach- 
ing and members of AERA's Division D (Measurement and Research Method- 
ology). Seventy percent of the participants entered the Institute with 
Limited "xperience in Bayesian statistics, with only 'wo individuals 
having no prior experience in the subject. The d.-siie to gain new 
skills or knowledge was the predc^minant rc'as(^n participants attended 
the Institute. 

One half of the participants lived wittiin 100 to 500 miles of 
the site of the Institute and two individuals lived within the immediate 
vicinity (less tlian 50 miles). Either a sense of immediacy or other 
personal or professional reasons is suggested as tlie dominant motiv-ation 
for the. five persons who traveled over 1,000 miles to attend. 

Table 1 
Description of Participants 
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Employing Ins ti tut ion 

College or Regional Center Federal or State Dept. Student Other 
University or Lab. of Education 

11 2 3 13 



Primary Professional Role 
Te-ichor Administrator Researcher Evluator or Developer Other 

9 14 5 2 



(Tablo 1 com . ) 



Ac ••cicTni , Dc;2;rcas 



li.'clu'Ior 's 



M.- s( -r 's 
9 



Doc tor-i tc 
5 



Membership in Divisions of AERA 

(B) Curricul'im & Objectives (C) Learning & (D) Mc.^.snremon f «Sc Resc-rch 

I ns t rue tion Methodolo^V 
1 4 14 " 



(H) School Rvilunrion & 
Prof^r.'im Development 



1 



( ) Nv^t presently a 
nu'inber of AERA 



Reasons for Attending the Institute 



Desire lo gain new 
skills or knowledge 
related to the topic. 
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Desire to improve 
exis ting skills or 
knowledge related 
to the topic. 



Desire to discuss 
problems related 
to the topic with 
experts . 



Other 



No 

Kxpcrionce 

9 



Prior Experience 



Limi "ed 
Experience 

14 



Conside rable 
Experience 



Great Amount 
of Experience 



less than 
50 miles 



Distance of home from Institute site 



50-100 
miles 



101-500 
mi] es 



501-1 ,000 
miles 



more than 
1000 miles 



10 




r.i;.^''-r^ of « I i> i ' J__ 

The prctL'iil L •-•.} i..' L ion iP:J:.:-TL ■ ■ >. :'c ndnin iiz : vrtd To pa r I", i c ip;:n L s 
.it lIil' oponing Si'h;sion ui lik' Instil' t.^ . Study macori^ils and a tentative 
.i|>(.nda were distributed to participi" r> j^riur to the Institute. The pr^.'- 
liniina. ry .;j7;enda was rL'or<^,an ir.od consii rably during thc^ four days as a 
result of the st<itus of the- computer. An announcement was made the first 
day that thc^ staff would be availa.bl. consultation during certain times 

when lectures were not scheduled. Tc ,unal time was scheduled during the 
four days for both class and individu.:L instructional purposes. 

Kvaluation Results 

C o^Qni t ive . 

The tabulation of the number of correct responses on the pre and post 
tests (repeated m(»asuri.\s) are includetj in Table II. 



Table II 

Frequency of Correct Answers for Pretest (n=10) and 
Posfest (n=15) on Cognitive Instrument, 



lies t ion # 


Pre Score 


Percent 


Post Score 


Percentage 


Percent 


1. 


6 


(607o) 


15 


(100%) 


+407, 


2 


4 


(40%) 


13 


( sr/) 


+4r/o 


•J a 


1 


(10%) 


3 


(20"/,) 


+ 107. 


b 


2 


(207o) 


9 


(607.) 


+407. 


c 


8 


(807.) 


14 


(9J7.) 


+137. 


d 


2 


(2070 


9 


(6070 


+40% 


c 


3 


(307o) 


4 


(27%) 


- 3% 


f 


2 


(20%) 


5 


(ST/o) 


-137. 




2 


(207o) 


6 


(40%) 


+20% 


h 


2 


(20%) 


2 


(13%) 


- r/. 


4 


9 


(907o) 


15 


(100%) 


+10% 


5 


7 


(70%) 


13 


(86%) 


+16% 


6 


8 


(80%) 


14 


(93%) 


+13% 


7 


0 


( 0%) 


5 


(33%) 


+33% 


8 


2 


(20%) 


14 


(9y/o) 


+7 3% 


9 


6 


(607„) 


13 


(86%) 


+26% 


10 


5 


(50/;) 


9 


(60%) 


+ 10% 


11 


8 


(807o) 


14 


(93%) 


+13% 


12 


6 


(607o) 


8 


(5 37o) 


- n 
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(Table H cont.) 



I 3 
14 
13 
1(. 
17 
18 
19 
20 



3 
5 
8 
7 
1 
b 
i 
8 



(3o: ) 

(80-.) 
(70.) 

do;',) 

(607) 

(30-;;,) 

(807.) 



3 

15 
10 
12 
13 
6 



(27:;) 

(337) 

(807,) 

(10 07,) 

(67/.) 

(80%) 

(8670 

(407o) 



- 3% 
-177 

07. 
+30% 
+5 77. 

+20:'. 

+56% 
-40% 



Pretest Mean 
Posttest Mean 
ChanRC 



22.6 (497) 
31.2 (68'.') 
8.6 (197) 



Analyses of v.nri-nce were performed between total pre and post test 
scores on this instn:- nt. The results are presented in Table III. 

The instrument was constructed by Victor Wilson after consultation 
with Dr. Novick concerning the content and objectives of the Institute. 
An item analysis was not computed and hence no reliability figures are 
reported. Each item of the cognitive test was counted as two points, 
with the exception of question no. 3, which was assigned a weight of's 
points, thus the 20-item test contained a total or 46 points. 

Significant differences were found at the .05 level, both between 
the pretest (0^) the same group on the posttest (O2) , and between the 
pretest (0^) and the unpretested posttest group (O3) . Therefore, within 
the limitations of the data reported, there was a significant over-all 
gain in participants ' understanding of the content of this Institute as 
measured by the 20 item instrument. No significant differences were 
found between the two posttest groups suggesting that there was no pre- 
test interaction effect for this instrument. 



Table III 



^ Analy.sos of Variance of Scores on Participant Instrument y/2, 

Coj;nitive Measure 



Pretest (0 > ~ ■^^) same group on post-test (0 , n = 8) 



sv 


SS 


df 


MS 


Treatment 


272.25 


1 


272.25 


Krror 


319.75 


7 


45.68 


Total 


592.00 


8 





Pretest n = 10) vs. unpretested group on post-test (0^, n = 7) 



sv 


SS 


df 


MS 


F p 


Treatment 


271.11 


1 


271.11 


5.2015 .05 


Krror 


781.83 


15 


52.12 




Total 


1052.94 


16 







Pretested post-test i^2^ n = 8) vs. unpretested post-test (O^j n = 7) 



sv 


SS 




MS 


F p 


Treatment 


3.10 


1 


3.10 


.1819 n.s 


f^rror 


221.30 


13 


1; .02 




Total 


224.40 


14 
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■ V t t i ludc> Towa rd i^av'.^si an Stalistu's 



Tlu' tabulation of responsos In i' i rL i c i[)ant Instnimont No. 3, iho 
a f foe tivc moasuro, is presented in Ta ■ 1 e IV. An analyses of variance 
performed on these results (Table V) i.'iowod a significant difference 
both betwecMT the pretest group (0^) and the same group on the posttest 
(O2) and between the pretest group (0) and the unpretested posttesL 
group (O3) . The analysis of viriance between the two posttest groups 
(pretest and unpretested) revealed n'a significant difference which 
again suggests no pretest interaction effect for the instruments. Thus 
it appears the Institute was successfu* in positi>'ely influencing parti- 
cipants' attitude toward Bayesian statistics. 

Par t Ic: ipan t Critique 

Participant ovL-raLl reaction to Mic Institute, as surveyed l)y the 
critique form (Participant instrument nuriiMcr A), was very positive. 
The tabulation of responses to most itLs.s fron: this instrument, along 
with a means and standard deviation, wh^'p aj)[:)rLipriate , are presented in 
Tabb'S VI, VII, and VIII. Some items have been deleted from this report 
as they pertain to the evaluation of the concept of Traveling Training 
Institutes rather than to the evaluation of this particular Institute. 

The vast majority of participants responded positively to the impor- 
tance of the Institute's topic for educational research and practice. 
All of the participants felt the topic should be treated again in future 
Institutes. The staff selected to present the topic was judged to be 
VL-ry well qualified. Nearly all the participants considered the date 
and Location of the Insitute convenient for them (data from Participant 
Instrument Number 1) . Seven of the 16 participants felt that the tui- 
tion fee was too liigh with 9 respondents reacting neutrally to the ques- 
tion. Stratification of responses according to thc^ total amount of 
money spent for each participant to attend the Institute revealed no 
meaningful relationships between the amount of money paid and the atti- 
tude towards the amount of the tuition fee, the topic, or usefulness of 
the Insitute. 

Participants were asked to record total expenses (including travel, 
tuition fee, living expenses, etc.) they had incurred by attending the 
Institute, as well as the amount of money they personally had to spend 
to attend the Institute (i.e., non-reimbursable expense items). Total 
costs ranged from $10 to $700, with a mean of about $370 in a median of 
$330. However, personal expenses ranged from $0 to $200, with a mean 
of about $30 and a median of $10. Only one individual was personally 
responsible for expenses over $100. 
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An-Mvju'S of V :riancc of Score: m Participant Instrument #3. 
Attitudes Toward vei^ian Statistics 



Analysis of variance bety^een the pretest group and the same group on the 
posttest (repeated measures) J 



SV 



SS 



df 



MS 



EMS 



MEAN 13122.00 

S (subjects) 233.00 

M (method) 80.22 

SM 112.78 



1 13122.00 

8 29.13 

1 80.22 

8 14.10 



18.0 
2.0 
9.0 
1.0 



5.69 



.05 



Analysis of^variance between the pretest group, and the unpretested group 
on the posttest. 



MS 



T rea tmen t 
Error 



217.39 
275.16 



1 

18 



217.39 
15.29 



14.22 



.01 



Analysis of variance between the two posttest groups (pretested and 
unpre tes ted) . 



SV 



SS 



df 



MS 



Treatment 



4.19 



4.19 



.1870 



n . s , 



I" r ror 



313.75 



14 



22.41 



1) One student guest took both tests and has been included. 

2) Students have been included. 
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Tabulation of Participant Rt .^onscs to Criticjuo Items 
Relating to P re-l ■ rt i tu tc Planning 



I tern 



How important do you feel the topi 
Institute is to educational resear-. 
eva lua ti on? 



very important 



1 2 3 

(5) (7) (1) 



of this 
' and/or 



4 5 
(0) (3) 



very unimportant 2.38 1.45 



How important do you feel the topi-, of this 
Institute is to educational practi-^e? 



very important 



1 2 
(4) (9) 



(2) 



(0) 



5 

(1) 



very unimportant 2.13 1.09 



Leaving aside the quality of instruction for 
the moment do you think the topi c L rented in 
this Institute should be treated again in future 
Ins titutes? 



definitely 



1 2 

(13) (3) 



3 

(0) 



4 5 
(0) (0) 



definitely not 



1.25 



.45 



18. In your opinion, the staff members were in 
general: 

very well-qualified 12 3 4 5 

(12) (3) (1) (0) (0) 

22. The meeting room facilities for the Institute 
were : 



very unqualified 1.3. 



.60 



excellent 12 3 4 5 

(2) (2) (5) (3) (4) 



poor 



3.38 1.26 



24. 



Considering what you have learned by attending 
this Institute (or any other benefits you have 
received from attendance), the tuition fee was: 

considerably too low 12 3 4 5 considerably 

(0) (0) (9) (4) (3) too high 



3.75 



.86 



Note: Frequencies nro shown in parentheses. 



P'l r t i t i ! Vi ;;p(Mi t tin- cc^ndncr of (.he Inst it at itself arc 
i'i><'i-(<fl ill i.ibU VH. Tlu' t (mU I'Tit v;: t\\c In.ititute was viewed as 
'lij'.hlv re! "int in Uic topi( ulv' rlisid \.>\l\\ sL<jfl ohjucLivos beinp, 
vrry ric.i!. .\ nLiiuriry of iIm p. i r ( i paiU s indicaLc^d the prerequisites 
MM" (In- (nsi itiilr wt-rc' clearly sL.itfd. f-^ip;}it of the 16 respondents 
In-lirvL'd ih' V i-nturud th«' Institute with more than adequate preparation 
and oulv on.' individual felt lie or sht' lacked the .adequate prerequisites. 

11k r Wert' mixed reaction to the planning and organization cf the 
Insti.iuf.'. This luiy be due in part lo the lack of differentiation between 

si.ilf';; vondiu t of the Institute and that of local coordination or 
It^'". i i' A pp r(v-; Lm.i t e 1 y one-half of the participants felt four days was 
iii:'.uli i i.i'* 10 ccwer the topic adequately. Tlie daily sessions were gcn- 

■ I" ! ! 1 V ( lM■|^i'K■^ed to he the right K'n|;t[i of time [■>ut most variations con- 
sid I t J the d.iil'v' scissions too shc^rt, 

"Mic nv.'TMll (ju.ility of the instruction in thcj Institutes was judged to 
i)r v/i r-. j:«uhI. Lt-etures were considered interesting and informative, with 
proper I) 1 L'ud of discussion and sufficient opportunity to ask questions. 
Tlu' ^.taj^ was reported to be very prepared and were willing to seek 

liie rc'iction participants to their instructional procedures. 

Responses to the open-ended questions were somewhat less positive 
than tliosr of tile structured items. Suggestions or recommendations for 
Lnip rf w in<', fulure Institutes are summarized in Table VIII. Items marked 
with an asterisk indicate items suggested by more than one respondent. 

Genc^ral Kv.il nation of the Institute 

Overall participant reaction to the Institute is summarized in 
Talkie FX. The majority of the participants believed that what they had 
le.irned from the Institute would be useful to them in their work. None 
of the participants considered their learning experience as having no 
iitiliiv to their work. Nearly all the respondents stated they would 

■ ittL-nd the Institute^ if they had to choose again and would recommend it 
to a eol li Mgut) . 



m 



- \ \ - 
Tablo. VLl 



Tabulation of Particlr-nl R^^sponses to Items 
on Conduct of tlu Institute 



1 1 em X s 

i. How relevant was the content of the lusti^'ute to the 

topic that was advertised? 1.31 .48 

highly relevant 1 2 3 4 5 not al all 

(10) (5) (0) (1) (0) relevant 

5. The staff's objectives for this Institute were: 1.81 .91 

very clear to 12 3 4 5 very unclear 

me (7) (6) (2) (1) (0) to me 

6. The planning and organization of thir, Institute was: 2.73 1.28 

excellent 1 2 3 4 5 ;:»oor 
(2) (6) (3) (2) (2) 

7. Overal, was the Institute long enoii,^li to cover tlie 

topic adequately? 3.44 1.03 

considerably 12 3 4 5 considerably 
too long (1) (1) (6) (5) (3) too short 

8„ As a rule, daily sessions were: 3.31 .79 

considerably 12 3 4 5 considerably 
too long (0) (1) (10) (3) (2) too short 

9. Do you feel ail necessary prerequisites were clearly stated 

in information you received prior to the Institute? (check ONE) 

12 Yes, chey were clearly stated 

0 No, they were ambiguous 

3 No prerequisites were listed 

10. Did you receive advance reading materials frcrni the Institute 
director(s) early enough to read them prior to ':he Institute? (Check ONR) 

14 Yes, materials came early enough 
0 No, materials came too late 
2 No materials were sent 

11. Do you think you entered the Institute with the appropri- 2.44 .81 
ate prerequisites or prior knowledge to make what you 

learned there of value to you? 

1 had more than 12 3 4 5 1 was seriously 
enough preparation (2) (6) (7) (1) (0) lacking in preparation 
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Table VTI (coni .) 



Tabulation of Partici]> -at Responses to Items 
on Conduct of the Institute 



1 1 em - s 

12. How often did the instruc t ion.i] procidurus tnkc into 
.uct-^unt variability in prior knowledge brouglit to the 

Institute by participants? 2.94 l.Oi 

always 1 2 3 4 3 never 
(0) (7) (5) (2) (2) 

13. Overall, the quality of ins t rue t ir a in this Institute 

was: 1.88 .7 

excellent 1 2 3 4") poor 

(5) (8) (3) (0) (0) 

14. Tiic instruction was generally: 

too lecture-oriented 12 3 4 5 too discussion- 

(0) (1) (15: (0) (0) oriented 2.94 .2: 

very interesting 12 3 4 5 very unintcrest- 

(7) (7) (2) (0) (0) ing 1.69 .7i 

very informative 12 3 4 5 very uninforma- 

(9) (6) (1) (0) (0) Live 1.50 .6 

15. Opportunities for asking questions and studying were: 

sufficient 12 3 4 5 insufficient 1.50 .8 

(10) ( 5) (0) (I) (0) 

16. The amount of work the staff required of you was: 3.44 .6 

far too much 1 2 3 4 5 far too little 

(0) (0) (10) (5) (1) 

17. Opportunities for vcu to interact with the siaff with 
respect to problems in your own work which relate to 

the topic were: 1.88 l.C 

sufficient 12 3 4 5 insufficient 

(6) (8) (1) (0) (1) 

18. In your opinion, the staff members were in general: 1.44 .7 



very well-prepared 1 2 3 4 5 very 

(11) (3) (2) (0) (0) unprepared 



t 



Tablr VI 1 (com.) 



Tabulation of ParticipanL Rjsponst'S to Items 
on Conduct of I he fnstitute 



I torn X s 

19. Did the St, iff Seek your roaction^; to their instructional 

procedures, scheduling, etc, dur xn^ the Institute? 2.25 1.06 

frequently 1 2 J 4 5 never 

(3) (7) (5) (0) (1) 

20. Did it appear to you that your reactions led to improve- 2,25 ,77 

frequently 12 3 4 5 never 

(4) (5) (7) .0) (0) 

21. The formal evaluation of this Inst:iLute by the "outside 

evaluation" (c.,>;. , those instrumenLs) was: 1.88 .89 

not nt all disruptive 1 2 4 5 very dis- 

(5) (8) (2) (1) (0) ruptive 



Notu: Frequencies are shown in parentheses, 
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Tabic VIII 



Pnr t icipan t sngpcstions for improving' ' lu- s.'ssion from responses to 



* Hdvo available p.ick.iges of sample; or r<'!\i data with hypothetical situ- 
ations to make tcnninal time more mo :'^.ingfnl 

Omit vali'e judgments relative to the merits of Bayesian vs Classical 
s ta tis tics 

Increase the pace of instruction 
More lectures and less terminal time 
Earlier social get-together 
More time for discussion 

List of participants and their areas of interest 
Conduct evening sessions 

Conduct lectures in the morning and terminal time in the evening 
" Improved lodging and meeting facilities 

More emphasis on tlie intiiitive meaning of formulas 
Condense Institute to 3 days 

A list of the order in which materials will be discussed 
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'I nhU r.X 

T:i!nil -It iiMi of Ta rt ic ipa;i t Rusjfonsos to Items on the 
General Eva lu/.i t ion c^f* Ihe Institute 



2 3. Do you anticipate tfiat wIkiC you 5iavt^ learned from 

the Institute will he useful to you in your own work? 

definitely 1 ! 3 4 5 del initely not 2.00 .73 

(4) (9) (3) (0) (0) 

27- If you were ahle to choose over again, would you attend 
tlj is Ins tl t:u te? 

(13) Yes No Xll Uncertain 

28. If this Institute were held again, would you ri'commcnd 
to a collea^^.ue tliat they attend? 

(14) Yes (0)_ No Uncrrtein 



Note: Frequencies are shown in parentheses 



v;j]u;jij(;n of AJ-RA Travelinr Training:, JnaLilutc 



"Bayesian SLaLisiics and interaciivo 
Comput. irig Systems" 

Directed by 

Melvin R. i^ovick 
American College Testing Program 

and 

Donald Meyer 
University of Pittsburgh 

January 12 - 15, 1973 
University of South Florida 



The ovnluati-on report of the second bayesian Training Institute 
is being presented in a somewhat abbreviated form. The evaluation 
design, administration, selectioii of participants, appendix, etc. that 
have remained constant and reportcid in the first evaluation report have 
been deleted. 

Institute Participants who Attended this Institute 

There were tx.enty six registered participants who attended the 
Institute. In addition, approximately ten students and faculty of the 
University of South Florida attended on a complimentary basis. 

The description of participants, Table I, portrays characteristics 
similar to those v;ho attended the first Institute. Differences occur 
in participants* wider membership in AERA divisions, although Division D 
remained as the predominate division. As a group, participants had less 
experience in the subject matter than those attending the University of 
Massachusetts Institute. This might be expected as the size of the 
class increases. Votb participants ti"avelled a greater distance to 
attend this Institute than those participating in the earlier session. 
This may have implications, for the concept of travelling training 
institutes in reducing cost.; and the importance of the site as an 
attraction to participants. 



Table I 



Description of Participants 



Employing Institution 



College or 
University 



Regional Center 
or Lab. 



Federal or State Dept. 
of Education 



Student 



22 



1 
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1 



Primary Professional Role 

Evaluator or 

I'cachur Administrator Researcher Developer Student 

16 2 ^1 3 1 



Academic De^^.rees 
Master * s Doctorate 
7 19 



Membership in DlN/isions of AERA 



(A) Administration 
3 

(D) Measurement & Research 
Methodology 
13 

(G) Social Context of 
Education 
3 

( ) Student 
1 



(B) Curriculum & 
Obj ect ives 
5 

(E) Counseling & Human 
Development 
2 

(H) School Evaluation & 
Program Development 
3 



(C) Learning & Instruc- 
tion 

3 

(F) History 



( ) Not presently a 
member of AERA 
10 



Reasons for Attending^ the Institute 



Dpsire to gain new skills 
or knowledge related to 
the to uc , 



Desire to improve 
existing skills or 
knowledge related to 
the topic. 



Desire to discuss 
problems related to 
the topic with experts. 



22 



0 



Prior Experience 

No Limited Considerable Great Amount of 

Experience Experience Experience Experience 

8 16 2 0 



ERIC 



2 



Distance of home from Institute site 



Loss than 101-500 500-1,000 more than 

50 miles miles miles 1,000 miles 

2 1 1 10 



Evaluation Results 
Co^nLt i ve 

The tabulation of the number of correct responses on the pre-and 
post-tests (repeated measures) are included in Table II below. As in 
the first Institute, a significant overall gain was recorded in participant 
understanding of the content of the Institute as measured by the twenty 
item test. 



Table II 



Frequency of Correct Answer*: 
Postest (n=24)^ on Co 



t;.sLi(jn // 


Pre 


1 


6 


2 


3 


3a 


1 


b 


3 


c 


8 


d 


2 


e 


0 


f 


1 


g 


0 


h 


1 


4 


10 


5 


5 


6 


6 


7 


1 


8 


4 


9 


10 


10 


2 


1] 


10 


12 


3 


13 


3 


14 


3 


15 


9 


16 


6 


17 


2 


18 


8 


19 


4 


20 


4 



tor Pretest (n=10) and 
.,ni t ive Instrument 



jre Post Score 



/, 




h 


60 


21 


88 


30 


19 


79 


10 


6 


25 


30 


4 


17 


80 


21 


88 


20 


7 


29 


0 


5 


21 


10 


5 


21 


0 


8 


33 


10 


7 


29 


100 


23 


96 


50 


13 


54 


60 


18 


75 


10 


13 


54 


40 


20 


83 


100 


23 


96 


20 


20 


83 


100 


21 


88 


30 


17 


71 


30 


5 


21 


30 


8 


33 


90 


22 


92 


60 


18 


75 


20 


16 


67 


80 


23 


96 


40 


20 


83 


40 


8 


33 



1 

Note: Two participants did not complete the post-test instrument. 



The analysis of variance performed between the total pre-and post- 
test scores on this instrument are presented in Table III, No significant 
differences were found between the two post- test groups suggesting that 
as in the previous evaluation, there were no pre-test interaction effects 
for this instrument. 



• 
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Table ill 



Analyses of Variance of Scores on Pariicipant Instrument //2, 

Cognitive Measure 





Pretest 


(OjL» ^ 


= ^ Ci^ ire 

— xv J vs. 


same group 


on posc^cesc vUo* n — mj 




O V 




C) o 


U ] 


rib r 


P 


Mean 




13676.45 


1 


13676 .45 




Treatment 




272.25 


1 


272.25 




S (Subjects) 




288.05 


9 


32.01 




M (Measures) 




451. 25 


1 


451.25 109,03 


.001 


SM 




3, ..;5 


9 


4. 14 






Pretest 


(Oi, n 


= 10) vs. 


unpretested 


group on post-test (O3 , n = 


: 14) 


SV 




SS 


df 


MS F 


P 


Treatment 




382.73 


1 


382.73 13.4525 


.005 


Error 




625.90 


22 


28.45 




Total 




1008.63 


23 








Pretested post- 


test (O2, 


n = 10) vs. 


unpretested post-test (O^, 


n = 14) 


SV 




SS 


df 


MS F 


P 


Treatment 




11.43 


1 


11.43 .4672 


n.s. 


Error 




538.40 


22 


24.47 




Total 




549.83 


23 







Attitude TQv;ard Bayesian StaListic :; 

The tabulation of responses Lj Pci rticipart Instrument No. 3, the 
affective measure, is presented in fable IV. -An analyses of variance 
performed on these results appear in Table V. Contrary to the results 
of the first Institute, no significant difference wats found between the 
pre-test group (0^) and the unpretested pcis*:-test group (0-^) , Also there 
was no significant difference found bet\/een the two post- test groups 
(0^ and 0^) . However a significant dliiereace did occur between the pre- 
test group (0^) and the same group on the post-test (02)- 
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Table V 



Arialyaes of Variance of Scores on Participant Instrument //3, 
Attitudes Toward Bavsian Statistics 



Analysis of variance between the pretest group (0-j_, n = 11) and the r.ame 
group on the posttest (O2, n = 11) (rej^eated measures). 



SV 



SS 



df 



MS 



Mean 13010.23 

S (subjects) 490.27 

M (measures) 204.05 

SM 204.4) 



1 

10 
1 
10 



13010.23 
49.03 
204.05 
20.45 



9.9799 



.05 



Analysis of variance between the pretest group (0-j^ , n = 11) and the un- 
pretested group on the posttest (0^ , n = 13) 

SV SS df MS F p 

Treatment 94.34 1 94.34 3.20 n.s. 

Error 647.62 22 29.44 

Total 741.96 23 



Analysis of variance between the two posttest groups (pretested (02> n=ll) 
and unpretested (O'-^y n=13) 

SV SS df MS F 2 

Treatment 26.57 1 26.57 .5636 n.s. 

Krror 1037.26 22 47.15 

Total 1063.83 23 



ERIC 



Participant Critique 

Participant overall reaction to the Institute, as surveyed by the 
critique form, was very positive. Tlic tabulation of responses to most 
items frori} this instrument, along ^nth a means and standard deviation, 
v;hcn appropriate, are presented in Tables VI, VII and VLII. 

As in the first institute, participants respo, .ed positively to 
the importance of the Institute's topic for educational research and 
practice. All but one of the participants felt the topic should be treated 
again in future institutes. The staff selected to present the topic was 
judged to be very well qualified. Fifteen of the twenty-four participants 
felt that the tuition fee was about right. A majority of the participants 
considered the meeting room facilities as poor. 

Participants were asked to record :ha Lotal expenses they had incurred 
by attending the Institute, as well a amount of money they personally 

had to spend to attend the Institute. Total cost ranged from $175 to $600 
with a mean of about $415 and a median of $400. However, personal expenses 
ranged from $0 to $400, with a mean of about $81 and a median of $50. 
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Table VI. 



Tabulation ot" Participant Responses to Critique Items 
Relating to Pre-Insultute Planning 



I te rn X s 

1. How important do you feel the topic of this 
Institute is to educational researc'i and/or 
evaluation? 

very important 12 3 4 5 very unimportant 1.75 l.V 

(15) (4) (2) (2) (1) 

2. How important do you feel the topic of this 
Institute is to educational practice? 

very important 12 3 4 5 very unimportant 1.88 l.V 

(13) (5) (3) (2) (1) 

4 . Leaving aside the quality of instruction for 
the moment do you think the topic treated in 
this Institute should be treated again in future 
Institutes? 

definitely 12 3 4 5 definitely not 1.33 .8 

(19) (4) (0) (0) (1) 

18. In your opinion, the staff members were in 
general : 

very well-qualified 12 3 4 5 very unqualified 1.13 .3' 

(21) (3) (0) (0) (0) 

22. The meeting room facilities for the Institute 
were : 

excellent 1 2 3 4 5 poor 3.92 .9 

(0) (2) (5) (10) (7) 

24. Considering what you have learned by attending 
this Institute (cr any other benefits you have 
received from attendance), the tuition fee was: 

considerably too low 12 3 4 5 considerably 3.29 .5 

(0) (1) (15) (8) (0) ^^Sh 



Note: Frequencies are shown in parentheses. 
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Participant responses to the r onduct of the Institute i tsel f are 
reported in Table VII. The conteni: of the institute was viewed as 
highly relevant to the topic advertised with staff objectives being very 
clear. A majority of the participants indicated the prerequisites for 
the Institute were clearly stated. Although ten of the twenty-four 
respondents believed they entered the Institute with more than adequate 
preparation, there were more participants in this Institute with a lesser 
amount of preparation than in the first Institute. Slightly more than 
one -half of the r2s[n>ndents indicated that the advance reading materials 
arrived too late for them to read prior to the Institute. 

The planning and organization of the Institute was considered to 
be very good. Approximately one-half of the participants felt four 
days were somewhat insufficient to cover the topic adequately. The daily 
sessions were generally considered to be of the right length of time. 

The overall quality of the instruction in the Institute was largely 
considered to be between vei.y good and excellent. Lectures vjere judged 
to be very interesting and informative, with a proper blend of discussion 
(although variation from the norm considered daily sessions heavy on lectures) 
and had sufficient opportunity to ask questions. The staff was reported 
to be very well prepared and generally willing to seek the reaction of 
participants to their instructional procedures. 
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Table VII 



Tabulation of Participant Responses to Items 
on Conduct: of t... : Instiiute 



i tem X s 

3. How relevant was the content of tln: Institute to the 
topic that was advertised? 

highly relevant 1 2 J A ) not at all 1.38 .71 

(17) (6) (0) (1) (0) relevant 

5. The staff's objectives for this Institute were: 

very clear to 1 2 3 4 5 very unclear 1.75 .79 

me (10) (11) (2) (4) (0) to me 

6. The planning and organization of this Institute was: 

excellent 12 3 4 5 poor 2.38 .92 

(3) (12) (7) (1) (1) 

7. Overall, was the Institute long enough to cover the 
topic adequately? 

considerably 12 3 4 5 considerably 3.58 .72 

too long (1) (10) (11) (2) too short 

8. As a rule, daily sessions were: 

considerably 12 3 4 5 considerably 2.79 .59 

too long ^0) (7) (15) (2) (2) too short 

9. Do you feel all necessary prerequisites were clearly stated 

in information you received prior to the Institute? (check ONE) 

18 Yes, they were clearly stated 
4 No , they were ambiguous 



2 No prerequisites were listed 



10. Did you receive advance reading materials from the Institute 

director(s) early enough to read Kheva prior to the Institute? (Check ONE) 

9 Yes, materials came early enough 



13 No, materials came too late 
0 No materials were sent 



11. Do you think you entered the Institute with the appropraite 
prerequisites or prior knowledge to make what you learned 

there of value to you? 2.71 .91 

I had more than 12 3 4 5 1 was seriously lack- 

enough preparation (g) (9) (5) (0) -'-^^ preparation 
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(Table V[I 



'. te rn 



in tcrm.s ol your back^.'^ round nr)d [Ut- .ar>ti. ion for 
Llifj InsLiluLc, the content oi Lhi5. inscitute was: 



excellent 12 3 4 5 
(13) (7) (2) (2) (0) 



p o r 



13. Overall, the quality of inst ruct ic»i i in Lhis 
Institute was: 



14, 



excellent 12 3 4 5 
(13) (7) (2) (2) (0) 

The instruction was generally: 



peer 



too lecture-oriented 



very interesting 1 



4 



too discussion- 



(2) (7) (15) (0) (0) oriented 



2 3 4 5 

(12) (8) (3) (J) (0) 

very informative 1 2 3 4 S 

(13) (7) (3) (I) CO) 



very uninteres- 
ting 

very uninforma- 
t ive 



15. Opportunities for asking questions were: 

sufficient 12 3 4 5 
(17) (6) (1) (0) (0) 

Opportunities for studying were: 

sufficient 12 3 4 5 
(15) (6) (2) (I) (0) 



insufficient 



insufficient 



16. The amount of work the staff required of you was 
far too much 



17. 



1 2 3 4 5 far too little 
(0) (1) (11) (9) (2) 



Opportunities for you to interact with the staff with 
respect to problems in your own work V7hich relate to 
the topic were : 



sufficient 12 3 4 5 
(8) (9) (5) (2) (0) 



insufficient 



18. 



In your opinion, the staff members were in general 

very well- 1 2 3 4 5 very un- 
prepared (16) (6) (0) (2) (0) prepared 



ERJC 
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(Table VII cont.) 



l^i. Uid the sLafi set»k your relictions i • their ins t rac L ionai 
|)ri)i'C'clurcs , scliedu 1 i , etc. durin v. Lho Institute? 

L'requi.'nt ly 12 3 4'. never 2.71 1.12 

(A) (6) (8) (5) ( . 

20. Did It appear to you that your reactions led to improve- 
ment? 

frequently 1 2 3 4 S never 2.71 1.30 

(3) (7) (6) (4) (::) 

21. The formal evaluation of this Institute by the "outside 
evaluation" (e.g., those instruments) was: 

not at all dis- 12 3 4 3 very dis- 1.70 .97 

ruptive (14) (3) (5) (1) (0) ruptive 



Note: Frequencies are shoi^m in parentheses. 



Responses to the open-ended questions were liomewhat less positive 
than those of the structured items. Suggestions or recommendations 
for improving future Institutes are summarized in Table VIII. Aspects 
of Che Institt„te that the participants felt were of consider ^,ble value 
are listed in Table IX. 
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Table V.' I I 



Participant suggestions for .'.laprovin;; the Institute 
from responses to open-ended c}uestions 



Fndividaully prescribed instruction anci :ri:erion referenced lectures lacked 
closure. SViould be condensed or shoulc )e omitted. (10) 

Real data should be available for analyr.is or participants should be instructed 
as to what kind of data could be anaiyztiJ and told to bring them to the Institute 
(9) 

Difficulty of computer being down - pl.-'u for alternative computing facilities (6) 
More interactive computing time (3) 
Schedule some free time (3) 

More examples and applications of Bayes'.ir tatistics (3) 

Provide for small group discussions and individuals to pursue their special 
needs (3) 

Hetter meeting facilities and non-smok i nv, in the room (2) 

An intuitive feel for Bayesian was developed only on the last day — reverse the 
structure (2) 

Omit football "game*' (3) 

Institute to be spaced over more days (2) 
Spend less time on the computer (1) 

Better orientation of group to the facilities, program and each other (I) 
There was too much assumed in practical applications (1) 

Allow the schedule to acconunodate more .'nformal discussion and reading or study (1) 

Shorter lectures (1) 

List of fellow participants (1) 

Advanced reading materials (1) 

Omit criticism of "classical statistics" (I) 

Omit lectures by Novick, ^^'erguson^ Savage and Isaacs (1) 

Institute should not be .leld over the weekend (1) 

Simultaneous lectures by staff would provide for alternatives (1) 

Analysis of questions from the group should be solved by lecturers (1) 
i^QI^-: Frequencies are shown in parentheses 

t^iyy 16 



able LX 



Aspects of the lnst.[tnC'. be I ievud be of 



I'ons i cIl' ;Mb [ I V3 ] lie 



Present at ions and lecliires ( i' ''vers (8) 

Use ol terminals ( \} 
Informal discussions with staff 
Introduction to Bayesian Point ol '/lew (3) 
Opportunity to put practice the l i.'.'ory (2) 
Techniques and strategy of anj>i\s' • [ i) 

Confirined my own implicit phllosop ly of "know your variable in research'* (1) 

Clarification of Bayesian Analysis (1) 

Di sues s ion of criterion rcrcmve r. eaf. hin^; (I) 

Critical attitude toward blin^ily man I pul at ive statistics (I) 
Note : Frequencies are shown in parentheses 

General Evaluation of the Institute 

Overall participant reaction to the Institute is summarized in Table X. 
The vast majority of the participants believed that what they had learned 
from the Institute would be useful to them in their work. Only one of 
che participants considered the learning experience as having no utility 
to his or her work. Nearly all the respondents stated that they would 
attend the Institute if they had to choose again and would recommend 
attendance at the Institute to a colleague. 




Table X 



Tabulation of Participant Re.suunses to Items on the 
General Evaluation ol' the Institute 



I tern x s 

23. Do you anticipate that what you have learned from 

the Institute will be useful to you in your own work? 

definitely 12 3 4 5 definitely not 1.67 .92 

(12) (10) (1) (0) (1) 

27. If you were able to choose over again, vould you attend 
this Institute? 

(20) Yes ( 2) No ( 2) Uncertain 

28. If this Institute were held again, would you recommend to 
a colleague that they attend? 

(20) Yes ( 2) No ( 1) Uncertain 



Note: Frequencies are shown in parentheses 
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Summary and Recommendations 

Participant overall reaction to the second Bayesian Institute was 
quite positive. The importance of the topic, qualifications and prepara- 
tion of the staff, quality of the instruction, and organization were all 
considered to be between good and excellent. A significant learning gain 
was recorded on the cognitive measure and nearly all the participants 
would recommend attendance at the Institute to one of their colleagues. 

Although participants generally reacted highly positively to the 
training Institute, many of their suggestions for improvement are worthy 
of careful consideration especially those items from the open-ended 
questions that were common to a number of participants. Specifically 
such areas as having sample data available for analysis, lectures on 
individually prescribed instruction and criterion referenced testing, 
back-up capability in the event of computer failure, and providing options 
in the schedule that might accommodate the variety of individual needs 
or interests of che participants. 

In addition to the participant suggestions, many of which I concur 
with, I would like to mention a few other items that are not directly 
related to the data garnered from the questionnaires. For the most part 
they represent my subjective opinions as well as informal conversations 
with a number of participants. 

Professor Savage's presentation on Sunday did not seem to be 
especially relevant to the topic of the Institute. A two hour *'book 
review*' of a general statistics book of readings may have been interesting 
and informative to some, but in light of the content to be covered in four 
days (and the respondents who felt more time was needed to cover the topic), 
aevoting nearly an entire morning of one day to such a lecture might have 
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been better utilized in other ways. 

Attention still must be given to securing better meeting facilities 
and insuring the operating status o i: the computer. I believe a solution 
to this latter problem will improve the relatively heavy amount of 
lectures during the first i*wo days (when the computer was down) and 
provide part icif^ants with an option of attending a lecture or working 
on the computer. 

I would still advocate implementation of some type of a daily formative 

evaluation mechanism. I think this was more important for this Institute 

where the larger number of participants formed a group that was more 

diverse in background, experience and interest. A final suggestion 

concerns the instructional staff being present at certain daily sessions. 

I think some of the advantages that might be realized are instructors knowing 

exactly v;hnt material V7as or was not covered, the possibility of answering 

or clarifying audience questions in alternative ways, insure that similar 

2 

notation is used (e.g., s - sum of squares or variance), and provide the 
lecturer with a type of formative evaluation for his presentation. 

The most noticeable difference I observed in this Institute from 
the first session was the more concentration placed on the content of 
Bayesian statistics vs. a "hard sell" philosophical approach. Having the 
social event (dinner) earlier in the week and the establishment of office 
hours I believe were generally welcomed by the participants. 
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EVALUATION OF AERA TRAVELING INSTITUTE NO, k 
'^Alternative Conceptions of Evaluation** 
Tampa, Florida January I8-I9, I973 



Evelyn Brzezinski 
Evaluation Center 
The Ohio State University 



This report serves as the second In the series cf reports which comprise 
the formative evaluation of the AERA Traveling Institute Series on ''Alterna- 
tive Conceptions of Evaluation.** That institute was conducted by 
Michael Gcriven, Daniel Stufflebeam, and Diane Reinhard. It is assumed 
that present readers will have the first report In this series (by Averill 
and Worthen) available to them, and therefore certain Information and 
attachments available in that report are not included here. 

The institute was held Thursday and Friday, January I8-I9, 1973, 
at the University of South Florida, Tampa, Florida. The remainder of 
this report Is divided into two major sections, description of the 
Institute and evaluation of the Institute. At the conclusion of the 
report, a summary and some recommendations for future Institutes are 
presented. 



Three sub-scctlons are Included In this part of the report. They 
are (1) pre- 1 nst I tute planning, (2) Institute participants, and (3) 
conduct of the Institute. 



Following recamendat Ions made by Averill and Worthen in the first 
report, a packet of materials was sent to each participant approximately 
ten days before the beginning of the Institute. Included in the packet 
were a tentative agenda, several papers written by Scriven and 



DESCRIPTION OF THE INSTITUTE 



Pre->lnstltute Planning 
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Stufflcbeam» and a memorandum from the staff in which it was suggested 
that r.he enclosed materials be read before arrival in Tampa. 

Copies of the application form submitted by all participants were 
available for the staff to study. From these forms, it was possible 
to obtain information regarding participants* type of employment, edu- 
cational history, and professional and scholarly interests. In addition 
on the application form each participant briefly described his interest 
in attending the institute and how its topic was related to any specific 
needs or objectives he might have. It appeared obvious to the evaluator 
that staff members were able to identify many of the participants (In 
terms of whire they came from and what their job was) upon being intro- 
duced to theri by name; but it is unclear how much the information 
contained In the application forms affected the conduct of the institute 
In other words, did Scriven use the particular examples he did in his 
presentation on, e.g., pathway evaluation because of certain things 
he might have read in participants' application forms? If so, then 
inclusion of that exercise on the application form seems worthwhile. 

Institute Participants 

The evaluator assumes that participants were chosen for the same 
reason as was stated In the Averill and Worthen report: "... on 
the basis of their expressed professional need for the content of the 
Institute, as indicated in a personal statement submitted with each 
application form (p. U)," Of kS applications submitted, 26 persons 
were chosen to attend the institute. Two of the 26 persons were unable 
to attend, so two replacements were chosen. A list of participants is 
given In Appendix A. 



A description of institute participants on several characteristics 
is given In Table I. As in the first institute in this series, insti- 
tutions of higher education and public school systems employ the 
majority of the participants. In Portland, the percentage of parti- 
cipants so employed was 80; in Tampa, however, the percentage dropped 
to 65. This group of participants was quite evenly split among teachers, 
researchers, and evaluators, while in Portland the evaluators comprised 
almost half of the group. In terms of highest academic degree held, 
the Tampa group was very similar to the Portland group: the majority 
of persons hold a doctorate. A greater proportion of the Tampa group 
rated themselves as having had a great amount of experience with the 
topic of the institute (31% of the Tampa group vs. 21% of the Portland 
group). Averill and Worthen were surpriseu that two thirds of the 
Portland participants travelled over 500 miles (one way) to attend 
the Institute, and half of them travelled over 1,000 miles (one way). 
That phenomenon occurred again In this institute; 88% of the parti- 
cipants lived at least 500 miles from Tampa, and 6^% of them lived 
over 1,000 miles from the Institute site* That so many persons would 
have travelled so far Is even more surprising when one realizes that 
the mean amount that participants spent personally (i.e., the amount 
not reimbursed by their employing Institutions) to attend the institute 
was about $97.00 

Conduct of the Institute 
Participants began arriving at the institute site at 8:30 Thursday 
morning, January l8th. As they registered, pre- i nst i tute evaluation 
instruments were distributed to the participants. Participants were 



Table 1 

Description of Participants (N = 26) 



Employing Institution 



Co ] 1 ege 
or 
Univ, 


Public 
School 
System 


Educ. 
R&D 
Center 


Regiona 1 Fed. 
Lab Gov^t. 


State Dept. 
of Educ. 


Student 


Other 


]k 


3 


2 


1 2 


1 


1 


2 








Primary Professional 


Role 






Teacher 


Adml n. 


Researcher Eva I uator 


Devel opc^ 


Student 


Other 


8 


2 


6 


7 


1 


1 


1 



Masters 



Highest Academic Degree 



Doctorate 
21 



Adml ni St rat ion 
(Div. A) 



AERA Dtvtslonal Membership 



Curri culum &- 
Objectives 
(DIv. B) 



Learning & 
I nst ruct i on 
(DIv. C) 



Measurement & 
Research Methodology 
(Olv. D) 



8 



11 



Counseling & Human Social Context School Evaluation fr 

Development of Education Program Development Not presently 

(Div. E) (Div. G) (Div. H) AERA member 

2 2 8 5 



Table I continued on next page 



Table 1 (cont.) 
Description of Participants (N « 26) 



Primary Reason for Attending Institute 



Desire to gain new 
ski i i s or knowledge 
related to the topic 



Desire to improve 
ex i st i ng skills or 
knowledge related 
to the topic 



Des i re to discuss 
problems related 
to the topic 
with experts 
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F, or Experience with Topic of Institute 




No 

Experience 


L i m i ted Cons i derab 1 e 
Expe r 1 ence Exper 1 ence 


Great Amount 
of Experience 


0 


8 10 


8 




Distance of Home from Institute Site 




Less than 
50 ml les 


50-100 101-500 501-1 .000 
mi les mi les ml les 


More than 
1 ,000 miles 


0 


] 2 6 


16 



free to mingle and meet each other and/or browse over numerous evalua- 
tion-related materials which had been placed together in one room. (A 
list of the resource materials available In Portland was provided In 
Appendix D of the Aver i i 1 -Worthen report; the same materials, with one 
or two additions, were available in Tampa as well.) At 9:30 a.m., 
participants and staff assembled in the meeting room and administrative 
details (e.g., agenda, plans for meals) were discussed. At S:kO, the 
substantive content of the institute was begun. 



Scriven first gave ah overview of alternative conceptions of evalua- 
tion and proponents of each conception. He then discussed pathway 
evaluation. Unlike the Portland institute, staff /part i cl pant discussion 
was present throughout all of the staff presentations. An animated 
discussion was in progress when, at 11:00, it was terminated so that 
relative adherence to the schedule could be maintained. After ^ five 

minute break, Stufflebeam began by referring to the articles which had 

1 

been sent to participants in the pre- i nst i tute package. He then 
presented seven general classes of problems faced by evaluators and, 
after lunch, discussed one of them the conceptual problem — in 
some detail. At 1:30 p.m., Relnhard spoke on the advocacy team 
approach to input evaluation. At 2:30, Scriven discussed meta-eval uat ion 
and goal-frse evaluation. A fifteen minute break was given at 3:15. 
Following that, a general discussion was held, and at 4:15 the group 
began to work on the Sand*»rs and McClellan evaluation materials (ref- 
erenced in the Aver i 1 1 -Worthen report). At flrst> the staff seemed 
unclear about how to proceed, but finally it was decided that the total 
group would work on one or two of the problems and later, perhaps, 
small groups would be formed to work on additional problems. Once 
discussion began, about half of the group of participants contributed 
comments. At 5:15i this activity ceased and after questions which 
participants wanted the staff to answer on Friday were written and handed 
in to staff, participants and staff left for dinner and the rest of 



I 

It would be useful to add an item to a post- i nst i tute evaluation 
instrument asking If the materials were read before arrival. 
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the evening. The 8:30 - 10:00 p.m. session scheduled for use of the 
resource table was not held. 

At 9:15 Friday morning, the group was divided into four subgroups 
for more work with the Sanders-McClel Ian problems. The group reconvened 
at 10:15 for subgroup reports. After a short lunch break, Stuff lebeam 
and Scriven discussed several of the topics which had been written down 
and turned in on Thursday (e.g., accountability, needs assessment, and 
criterion- vs. norm-referenced testing). This discussion continued until 
2:^5 p.m., at which time the final evaluation instruments were administered. 
After that, the institute adjourned. 

EVALUATION OF THE INSTITUTE 

The methods used to evaluate the institute and the results of that 
evaluation are contained in the remainder of this report. 

E valuation Plans and Procedures 

The design for the evaluation of this institute was produced by 
Marilyn Averill and Blaine Worthen of the laboratory of Educational 
Research, University of Colorado. All Instruments u^ed to evaluate the 
Institute were designed by them; in two instances, the present evaluator 
modified items to pertain specifically to the content of the institute 
in Tampa. As in the Portland institute, there were four instruments 
to be completed by participants and one instrument to be completed by 
staff. The four participant Instruments contained items requesting 
(1) demographic data (Instrument #1), (2) self-reported cognitive out- 
comes (Instrument ^2), (3) affective outcomes (instrument #3), and 
{h) evaluative data about the content and conduct of the institute 
(Instrument Hk) . The staff Instrument likewise contained questions 
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of an evaluative nature about the conduct of the institute. 

Although Averill and Worthen had suggested in their report of 
the Portland institute that a cognitive achievement test might be a 
worthwhile addition to the evaluation instruments, one was not developed 
for use In Tampa. This evaluator is ambivalent about the usefulness 
of such an instrument. From reading Stuff lebeam's goals for the 
institute (Averill & Worthen, p. 4), it seems that the overall objec- 
tive of the institute is to make participants aware of evaluation 
problems and issues and not to reach specified cognitive outcomes. If 
this is the case, then it seems that a follow-up study of the type 
planned is as useful, if not more useful, than a cognitive achievement 
test. 

Instrument Administration and Analyses 
Instrument #1 (demographic data) was administered to each parti- 
cipant prior ^to the beginning of institute activities. At the same 
time, 13 of the 26 participants were chosen randomly to respond 
Instruments #2 and ffi (cognitive and affective measures). Instruments 
//2, ^3, 5ind and the staff in*.trument were administered to the total 
group at the conclusion of the institute. The design of the evaluation 
appears below: 

R 0, X O2 

R X 

where 0| is the 13 responses to Instruments ^2 and #3 before the insti- 
tute, O2 is the responses to Instruments #2 and #3 from those same 13 
participants, and 0^ Is the responses to Instruments ff2 and #3 made by 
the remaining 13 participants at the end of the institute. 
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The following analyses were performed: 

(1) Descriptive statistics (tabulations with means and standard 
deviations where appropriate) for all Instruments. 

(2) One-way analysis of variance between total pre- and post- 
Institute scores on instruments #2 and #3, for the following 
cofnpar i sons : 0| - 0^, O2 - 0^. 

(3) Analysis of variance (repeated measures design) between total 
pre- and post- inst i tute scores on Instruments #2 and #3 for 
the t-omparison 0] - O2 . 

The analyses were performed by Marilyn Averill at the Laboratory of 
Educational Research, Universi ty of Colorado. 

Eval uat ion Resu 1 ts 
The results of this evaluation are organized under the three 
evaluative participant Instruments and the one staff instrument which 
were used in the evaluation, 
Self-Report of Knowledge (Instrument ^2) 

Tabulations of pre- and post- 1 nst i tute responses on the self- 
report of knowledge are presented in Table 2. Analyses of variance 
were performed between total pre- and post- I nst i tute scores on this 
f.-strument; results are shown In Table 3. Significant differences at 
the .001 level were found between the pre- i nst i tute scores (0|) and 
the post- i nst i tute scores for the same group (O2), and at the .05 level 
between the pre- I nst 1 tute scores (0|) and the unpretested post- i nst i tute 
scores (Oj). Since there was no significant difference between the two 
post- J nst i tute scores, it can be assumed that a pretest sensitization 
effect for this instrument was not a factor. Participants apparently 
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Table 3 



Analysis of Variance: Scores on Self-Report 
of Knowledge (Participant Instrument #2) 





Compa r i son : 


Ox 


- O2 (pre- i nst i tute , 


11 = 13, vs. same 13 on pes':- i nst i tute) 


SV 




SS df 


MS F p 


Subj ect s 




923.85 12 


76.99 


Treatment 




h08.0k 1 


4o8.0i» i47.78 .001 


Res • dua 1 




102.^ 12 


8.5^+ 




Comparl son: 


Ol 


- (pre- i nst i tute, 


n = 13, vs. unpretested post- i nst i tut e 








n = 12->) 


SV 




SS df 


MS F p 


Treatment 




215.!^ 1 


215.1^ ^.52 .05 


Error 




1055.90 23 


^47. 65 


Total 




1311.0^4 2k 






Compa ri son: 


02 


- 0^ (pretested post 


-institute, n «= 13t ^s. unpretested 






post- i nst i tute, n » 12^) 






SS df 


MS F p 


Treatment 




26.26 1 


26,26 .70 n.s. 


Error 




867. 7^+ 23 


37.73 


Tote ! 




89^+. 00 2k 





One person from this group failed to complete the pos t- i nst i tute instru- 
ment, so total N « 25 rather than 26. 



perceived themselves as having significantly increased their understand- 
ing of evaluation-related topics as a result of the institute; but with 
no performance data on cognitive items to measure the validity of thii. 
sel f -reported knowledge, it is impossible to say whether the partici- 
pants' perceptions are accurate or not. It can be said only thot parti- 
cipants f e 1 t they increased their knowledge as a result of the institute. 
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Attitudes toward Educational Evaluation (Instrument H'i) 

Tabulations of responses to Participant Instrument ^"3, the affective 
measure, are presented in Table k. Analyses of variance performed on 
these results (see Table 5) indicate that there was no pretest sensiti- 
zation effect caused by administering the instrument to participants 
before the institute, nor were there any significant differences between 
pre- and post- i nst i tute scores for either of the comparisons of interest 
(C| - O2 and O2 - Cj). The puzzling results found in the evaluation of 
the Portland institute (when the pre- i nst. i tute administration of the 
instrument seemed to function as a retardant to affective growth, see 
p. 15 of the Aver i 1 1 -Worthen report) were not duplicated with this group 
of participants. On the whole, the persons in Tampa had positive 
attitudes toward evaluation both before and after the institute. 
Critique Form (Instrument #4) 

A summary of responses to most items on the critique form Is pre- 
sented i Tables 6-9. As mentioned in the Aver i 1 I -Worthen report 
(p, 19), some items were deleted from this formative report because 
they pertain to evaluation of the concept of traveling institutes 
rather than to the evaluation of this particular institute. The 
remaining critique form items have been grouped into four logical 
clusters, and the results in this section have been subdivided into 
four corresponding sections. In the report of the Portland institute, 
some item responses were analyzed after stratifying un (I) total amount 
of money spent by participants to attend the institute and (2) f/nploy- 
i ng institution; those analyses did not prove to be particularly 
ineanfnqful or useful, and so they were not performed here. 
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Table 5 



Analysis of Variance: Scores on Attitudes toward 
Educational Evaluation (Participant Instrument #3) 





Comparison; 


0, - O2 (pre- 


inst i tute, 


n = 13, vs. same 13 on post - i nst i tute) 


SV 


ss 


df 


MS F p 


Subj ects 
Treatment 
Res i dual 


369.46 
.15 
36.85 


12 
1 

12 


30.79 

.15 .05 n.s. 
3.07 




Compar i son: 


0, - O3 (pre- 


1 nst i tute , 


n = 13, vs. unpretested post- i nst i tute 

n = 11*) 


SV 


ss 


df 


MS F p 


Treatment 

Error 

Total 


2.36 
308.14 
310.50 


1 

22 
23 


2.36 .17 n.s. 
14. CI 




Comparison: 


O2 - O3 (pretested post- 


■institute, n ■ 13, vs. unpretested 
post- 1 nst i tute, n » Ih'") 


SV 


SS 


df 


MS F p 


Treatment 
Error 
Tota 1 


1.35 
335.99 
337.3^ 


\ 

22 
23 


1.35 .09 n.s. 
15.27 



* Two persons from this group failed to con^plete the post- i nst 1 tute instru- 
ment, so total N « 24 rather than 26. 
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Evaluation of Pre- I nst i tute Planning . Participants were asked to 
record their impressions of pre- i nst i tute planning in areas related to 
the topic, staff, facilities, fee structure, location, and date. Parti- 
cipant responses to these questions are summarized in Table 6. As with 
the Portland group, participants responded very positively to the items 
relating to Ihe Importance of the institute topic and to the qualifi- 
cations of the staff. 

Participants generally felt that the tuition fee was a reasonable 
charge ($90 for AERA members, $100 for non-members). They were ssked 
to record the total expense (Including travel, tuition fee, and living 
expenses) Incurred by attending the institute, as well as the amount 
of money they spent personally (I.e., non*re imbursabl e expense Items) 
to attend the institute. Total costs ranged from $0 to $330, with a 
mean of about $2^5 and a median of $300. Personal expenses ranged from 
$0 to $325, with a mean of about $97 and a median of $10 (obviously, a 
few persons paid quite a bit more of their own money than most partici- 
pants did to attend the institute, thus causing the mean to be so high 
in relation to the median). 

2 

In response to two questions which appeared on Instrument #1, it 



2 

10. If this session were not available now, but was offered instead as 
a session for a similar amount of time either Immediately before or 
•ftcr the convention In New Orleans In February, would you choose 
to attend the session there? (Assume registration fees, etc., 
remained constant.) 

i i Yes 15 No 



11. As far as you now know, do you plan to attend the AERA Convention in 
New Orleans this year? 

16 Yes 10 No 



ERIC 
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Table 6 



Tabuletlon of Participant Responses to Critique Items 
Relating to Pre- i nst i tute Planning (N « 24^0 

Iteg X _s_ 

J. How Important do you feel the topic of this Institute 

Is to educational research and/or evaluation? 1.29 .86 

very important 1 2 3 ^ S very unimportant 
(20) (3) (I) 

2. How important do you feel the topic of this Institute 

is to educat ional practice? 1.63 1.0) 

very Important 12 3^5 very unimportant 
(15) (5) (3) (1) 

k. Leaving aside the quality of instruction for the moment, 
do you think the topi c treated in this Institute should 
be treated again in future Institutes? 1.17 .48 

definitely 12 3^5 definitely not 
(21) (2) (I) 

19. In your opinion, the staff members were in general: I.I7 .82 

very wel 1 -qual i f led 12 3^5 very 

(23) (1) unqualified 

23. The meeting room facilities for the institute were: 1.63 .97 

excel lent 12 3^5 poor 

(13) (5) (4) (1) 

25* Cons iderl ng what you have learned by attending this 
Institute (or any other benefits you have received 
from attendoncft), the tuition fee was; 3 21 .66 

considerably 1 2 3^ 5 considerably 
too low (2) (16) (5) (I) too high 

^'f Two persons failed to complete this instrument, so total N « 24 rather 
than 26. 

NOTE: Frequencies are shown In parentheses. 



ERLC 



can be seen that over half of the participants planned lo atlend the AERA 
annual meeting in New Orleans one month after ti.e Tampa institute; but 
fewer than half of the participants would have been willing to attend 
the institute as a pre- or post-session to the annual meeting. This 
perhaps suggests (and answers to open-ended questions on this same 
instrument confirm) that it is relatively easy for a person to get away 
from his prof esss iona 1 duties a few days at a time, but much more diffi- 
cult for him to be gone from his job for a week at a time (as attendance 
at the annual meeting and a pre- or post-session would have required). 

Most participants rated the meeting rooms as excellent or good. 
Sun and bugs at times proved distracting, but on the whole the facili- 
ties were more than adequate (and lovely to look at, as welll). 

Evaluation of Conduct of the Institute , Participants were asked 
to respond to questions dealing with the conduct of the institute itself, 
including concerns for the success of its activities, its content 
validity, its objectives, and the instruction. Summaries of responses 
to those it^s appear in Table 7. 

Generally speaking, responses to this set of items were more 
positive than from the Portland group. This was especially true in 
those items relating to the clarity of objectives (item 5), the plan- 
ning and organization (item 6), the information regarding prerequisites 
and advance preparation (items 9 and 10), the quality of instruction 
(items 13 and and the preparation of the staff (item 19). 

Evaluation of I nstitute Activities . Item 30 was concerned with 
specific activities during the institute. Participants were asked to 
respond to that Item In terms of how valuable the activities had been 
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Table 7 

Tabulation of Participant Responses to Critique Items 
on Conduct of the Institute (N = 2^^?) 



Item X 



3. Ho'w relevant was the content of the Institute to the 

topic that was advertised? 1.50 .78 

highly 1 2 3 k 5 not at al 1 

re 1 eva nt ( 1 6) {k) {k) re 1 eva nt 

5. The staff's objectives for this Institute were: 2.21 .72 

very clear 12 3^5 very unclear 

to me (3) (1^) (6) (1) to me 

6. The planning and organization of this Institute were: 2.04 .95 

excellent 12 3^5 poor 

(7) (11) (5) (1) 

7. Overall J was the Institute long enough to cover the 

topic adequately? 3.^3 .73 

considerably 12 3^5 considerably 
too long (2) (10) (10) (1) too short 

8. As a rule> da i 1 y sessions were: 2.96 .36 

considerably 1 2 3^5 considerably 
too long (2) (21) (1) too short 

9. Do you feel all necessary prerequisites were clearly 
stated In Information you received prior to the Institute? 
(check ONE) 

] 7 Yes ; they were clearly stated 
3 No> they were ambiguous 
2 No prerequisites were listed 



10. Did you receive advance reading materials from the Institute 
director(s) early enough to read them prior to the Institute? 
(check ONE) 

20 Yes, materials came early enough 
2 No, materials came too late 
1 No materibls were sent 



ERIC 
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Table 7 (cont.) 

Tabulation of Participant Responses to Critique Items 
on Conduct of the Institute (N «= 2^^'0 



I tern 
11. 



12. 



Do you think you entered the Institute with the appro- 
priate prerequisites or prior knowledge to tnake whcjt 
you learned there of value to you? 2.00 

I had more than 1 2 3 5 I was seriously 

enough preparation (7)(n) (5) (1) lacking in preparation 



In terms of your background end preparation for the 
Institute, the content of this Institute was: 



3.00 



too 

el ementary 



1 

(2) 



2 3 
(1) (17) 



(3) 



5 

(1) 



too 

advanced 



13. Overall, the quality of Instruction in this Institute 
was: 



1.58 



excel lent 



1 

(12) 



2 

(10) 



3 

(2) 



5 



poor 



\k. The I nstruct ion was generally 

1 



too 1 ecture- 
or lented 



2 3^ 
(6) (15) (3) 



very 12 3^ 

interesting (16) (8) 

very 12 3^ 

Informative (13) (8) (3) 



too discus- 
sion-oriented 

very 

uni nterest i ng 
very 

un i nformat I ve 



15. Opportunities for asking questions were: 

sufficient 12 3^5 
(16) (5) (3) 

16. Opportunities for study i ng were : 

sufficient 12 3^5 
(5) W (6) ik) ik) 



2.88 
1.33 

1.58 
1.46 



insufficient 



insufficient 



17. The amount of work the staff required of you was: 

far too much \ 2 3 ^ 5 far too little 

(1) (12) (9) (2) 



2.91 



3.50 



.83 



.83 



.65 



.61 
M 
.72 
.72 

1.^1 



.72 



ERIC 
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Table 7 (cont.) 



Tabulation of Participant Responses to Critique Items 
on Conduct of the Institute (N » 2^*'') 



Item X _s_ 

18. Opportunities for you to interact with the staff with 
respect to problems in your own work which relate to 

the topic were: 2.92 1.21 

sufficient 12 3^5 insufficient 
Ik) {k) (8) (6) (2) 

19. In your opinion, the staff rnembers were in general: 1.^2 .65 

very well- 12 3^5 very 
prepared (16) (6) (2) unprepared 

20. Did the staff seek your reactions to their instructional 
procedures, scheduling, etc., during the Institute? 1.88 .95 

frequently 12 3^5 never 

(TO) (9) (3) (2) 

21. Did It appj£5r to you that your reactions led to improve- 
ments In the instructional procedures, schedules, etc.? 2.21 .93 

frequent ly I 2 3 k 5 never 

(6) (9) (7) (?) 

22. The formal evaluation of this Institute by the "outside 
evaluator" (e.g., those instruments) was: 1.46 .78 

not at all ' 2 3 k 5 very 
disruptive (17) (3) (4) disruptive 

Two persons failed to complete this Instrument, so total N « 24 rather 
than 26. 

NOTE: Frequencies are shown in parentheses. 
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to them. A summary of their responses is presented In Table 8, 

Activities; rated as most valuable included Stuff 1 ebeam* s presenta- 
tion on problem^ ^^ced by eva i uato'^^i ; Scriven's presentations on goal- 
free and pathway evaluation, and the general question and answer 
sessions wit^ staff. The low rating achieved by "Individual reading: 
Use of reference table*' may be attributed to the fact that no one 
specific time was allocated for this purpose, the Tliursday evening 
sesbion having been cancelled. (To be truthful, it was not so much 
cancelled as that it simply failed to materialize. The institute site 
was quite distant from the motel where most participants were staying, 
and there was little inclination for participants to return to the 
meeting rooms after dinner, espec ial 1 y when it involved a 20-minute 
drive.) However, one must remember that the reading session scheduled 
in Portland was rated as not particularly useful, so apparently a 
scheduled time for reading (at least at the beginning of the institute) 
is not the answer to the problem. 

As 5n Portland, the Sanders-McClel Ian workbook examples were not 
rated as particularly valuable. However, in the opinion of the eval- 
uator, this is the act i v i ty where the staff seemed least prepared and 
organized. It would be difficult for the evaluator to determine whether 
the low rating is intrinsic to the materials or to the seeming con- 
fu:;ion at the beginning of the activity. 

Reinhard's discussion of advocacy teams again was not rated par- 
ticularly we 11, relatively soeaki ng. It was suggested in the Aver i 1 1 - 
Worthen report that this might have resulted from the fact that hers 
was the only presentation during which participants consistently asked 
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Table 8 

Tabulation of Participant Responses to Critique I terns on the 
Evaluation of Institute Activities (N 2k''-) 



30. How would you rate the vatue of each of the following aspects of this 
Inst I tute? 

(Please circle ONE for EACH report) 







no opportunity 
to Judae 


very 
valuable 


worthless 


X 


s 


a • 


Scrivpn: pathway evaluation 
(Thu'-bday morning) 


NA 


1 

(7) 


2 

(13) 


3 

(3) 




5 


1 .83 


.65 


b. 


S tuf f lebeam: problems, CIPP 
model (Thursday morning) 


NA 


1 

(13) 


2 

(7) 


3 

(^) 




5 


1 .63 


.77 


c. 


Reinhard: advocate team t-echnique 
(Thursday afternoon) 


NA 


1 

(3) 


2 

(7) 


3 

(10) 


k 

(3) 


5 

(1) 


2.67 


1 .01 


d. 


Scriven: goal-free evaluation 
(Thursday afternoon) 


NA 


1 

('2) 


2 

(10) 


3 


k 

(2) 


5 


1 .67 


.87 


e . 


Workbook examples 
(Thursday afternoon) 


NA 


1 

(3) 


2 

(8) 


3 

(7) 


k 

(5) 


5 

(1) 


2.71 


1 .08 


f . 


Individual reading: use of reference 
table (Thursday evening) 


NA 


1 

(1) 


2 

(1) 


3 

(5) 


k 

(!) 


5 

(6) 


3.71 


1.33 


g. 


"Special topics" presentation 
(FrI day morn i ng) 


NA 


1 

(M 


2 

(10) 


3 

(5) 


k 


5 

(1) 


2.20 


.95 


h. 


Discussion of individual problems 


NA 


1 

(3) 


2 

(9) 


3 

(5) 


k 

(1) 


5 


2.22 


.81 


i • 


General question and answer 
sessions with staff 


NA 


1 

(10) 


2 

(12) 


3 

(2) 


k 


5 


1 .67 


.64 


\ 

J • 


Informal discussion with peers 


NA 


1 

(8) 


2 

(11) 


3 

(5) 


k 


5 


1.88 


.7'* 



Two persons failed to complete this instrument, so total N = 2^4 rather than 
26. 

NOTE: Frequencies are shown in parentheses. 



questions (rather than waiting until the presentat i on was completed), 
thus possibly causing the presentation to los.* its continuity. As 
mentioned earlier in this report, participants in Tampa interrupted al I 
the presentations to ask questions; Reinhard's presentation was no 
exception. So apparently that was not a major cause of the low 
rating. The evaluator has another observation, however, which may 
suggest a cause for the low rating. Reinhard was the person who was 
responsible for all the administrative/secretarial details during the 
institute (e.g., making announcements about meals, handing out regis- 
tration materials, monitoring the materials on the reference table); 
Stufflebeam and Scriven were much more the "p^'of essors" which, of 
course, they are. However, the difference in status between Reinhard 
and the other two staff members appeared quite obvious to the evaluator; 
and it may have to the participants too, causing them to rate her 
presentation less favorably simply because she is new in the field. 
One would expect Reinhard's ratings to improve as she becomes more 
sure of herself. Although it may be inappropriate to compare ratings 
of the Portland group with the Tampa group on an isolated item, it is 
true that the mean rating for the advocacy team presentation was 3.33 
in Portland and 2.67 in Tampa. 

Responses to open-ended questions on the instrument maintained the 
positive attitude seen in Table 8. The only recurrent (3-5 times) 
negative comment was that participants would have benefitted from more 
practical applications of the concepts discussed (perhaps through 
s imulat ions) . 

General Evaluation of the Institute ^ Participants* overall 
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reactions to the Institute are presented in Table 9* The majorfty of 
the participants believed that (0 what they had learned from the insti 
tute would be useful to them in their work, (2) they would attend the 
Institute if they had to choose again, and (3) they would recommend 
the Institute to a colleague. 

Table 9 

Tabulation of Participant Responses to Items on the 
General Evaluation of the Institute (N ^ 2^*) 

I tern 

2^. Do you anticipate that what you have learned from 

the institute wM 1 be useful to you in your own work? 

definitely 12 3^5 definitely not 
(17) (6) (0 

28. If you were able to choose over again, would you 
attend this institute? 

(22) Yes (0) No (2) Uncertain 

29. If this institute were held again, would you reconnmend 
to a colleague that he attend? 

(21) Yes (0) No (3) Uncertain 

Two persons failed to complete this instrument, so total N = 24 rather 
than 26. 

NOTE: Frequencies are shown in parentheses. 
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SUMMARY AND RECOMMENDATIONS 

In summary, it may be said that the secunu "Alternative Conceptions 
of Evaluation'* institute was quite successful probabl y more so than 
the first Institute in Portland, which is to be expected. The demand 
for the institute seems great (as the number of applicants would imply) 
and the reactions of the oarticipants seem very positive (as the eval- 
uation results reported here would imply). It appears, then, that 
th^s institute series should be continued. 

Several recommendations were made by Averill and Worthen at the 
conclusion of their report on the Portland institute. Some of those 
recommendations were heeded; some no longer seem warranted on the basis 
of the "fampa data. A few, however, are still considered by this 
evaluator to be good recommendations for future institutes. They 
are presented below. 

1. "Information about participants' prior training and experience 
in evaluation should be collected and reviewed by the Institute 
directors to assure an appropriate match between participants' back- 
grounds and the content and activities of the Institute (Averill & 
Worthen, p. 38)." This point was discussed earlier in this report, 

see p, 2. 

2. "Internal evaluation should be emphasized more and external 
evaluation of programs emphasized less in subsequent programs (unless 
different types of participants are selected) (Averill & Worthen, p. 40), 
On several of the demographic characteristics, this group of partici,- 
pants did seem somewhat different from the participants in Portland. 

The same recommendation holds, however, based on responses to open- 
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ended questions and informal comments to the evaluator. It would seem 
that participants* requests for more practical applications of the 
concepts they are learning would aUeviate this problem somewhat. 
Perhaps the Sanders-McCle 1 !an materials could be emphasized more, since 
they do seem to be about day-to-day problems with which most partici- 
pants might be expected to be familiar; or If the institute staff does 
not feel that these materials are adequate, perhaps other sources could 
be searched to find practical situations in evaluation and the problems 
they entail that might be discussed more substantially during the 
institute. 



Append ix A 
Participants at Tampa Institute 



EVALUATION OF AERA TRAVELING INSTITUT 
"Alternative Conceptions of Evaluation 

Tucson , Ari zona 
March 28-29, 1973 



Marilyn Averill 



Laboratory of Educational Research 
University of Colorado 



/ 

This report is an evaKatio.i of the third "Alternative Conceptions 
of Evaluation" Traveling Institute presented by Diane Reinhard, Daniel 
Stufflebeam, and Michael Scriven in Tucson, Arizona, on March 28 and 29, 
1973, Descriptions of the evaluation desigp and of general Institute 
organization are available in the first evaluation report of this 
series, and will not be repeated here. 
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Institute Participants 

Nineteen participants attended the Institute in Tucson; their 
characteristics are described in Table 1, They vary more with respect 
to employing institution and professional role than did those attending 
the first Institute in Portland, although the majority (67%) are still 
employed by institutions of higher learning or public school systems. 
All but one participant hold graduate degrees, and 14 hold a doctorate. 
AERA divisional memberships vary considerably, with the largest numbers 
of participants belonging to School Evaluation and Program Development (10) 
and Measurement and Research Methodology (7), 

Participant experience with the topic of the Institute varied equally 
over limited, considerable, and great amounts of experience. Reasons for 
attending the session also varied considerably. 

As in Portland, most participants travelled long distances to attend 
the Institute? over half travelled more than 1,000 miles. This supports 
the idea that interest and professional need seem to attract more partici- 
pants than does regional convenience; however, it may also suggest that 
Institute sites have not been chosen with regard to regional centers of 
professional interest. It is possible that sites such as Chir.?go or 
Los Angeles would attract equal numbers and reduce participant travel 
distances. 



Table 1 

Description of Participants 





Employing Institution 




College or 
University 


Public School Regional 
System Laboratory 


State Department 
of Education 


8 


5 1 


2 


Student 


Other 




1 


1 






Primary Professional Role 




Teacher 


Admi nis trator Researcher 


Evaluator 


4 


1 3 


6 




Other 




2 


3 






Academic Degrees 




Bachelor's 


Master * s Doctorate 




1 


4 14 






Membership_in Divisions of AERA 




(A) Administration (B) Curriculum & (C) Learning & 

Objectives Instruction 


(D) Measurement . 
Research 
Methodology 


1 


3 4 


7 


(E) Counseling & 
Human 

Development 


(G) Social Context School Evalu- Not presently 
of Education ation and Program member of 
Development AERA 


1 


1 10 


3 



3 



Table 1 (continued) 
Description of Participants 



Reasons for Attending the Institute 

Desire to gain new Desire to improve Desire to discuss 

skills or knowledge existing skills or problems related 

related to the topic knowledge related to the topic 

to the topic with experts 



Prior Experience 

No Limited Considerable Great Amount 

Experience Experience Experience of Experience 



Distance of Home from Institute Site 

less than 50-100 101-500 500-1,000 more than 

50 miles miles miles miles 1,000 miles 

4 0 4 4 12 



Conduct of the Insti t ute 

Institute registration began at 8:30 Wednesday morning; preliminary 
evaluation instruments were also administered at this time, and supple- 
mental materials distributed. Most advance materials had been mailed to 
participants prior to the Institute. A resource table was available 
throughout the two days. 

The tentative agenda had to be revised considerably, due to Scriven's 
da> late arrival. However, presentations began as scheduled at 9:30; 
Stufflebeam began with the CIPP Model and problems faced by evaluators, 
followed by Diane Reinhard on Advocate Team Techniques. Participants 
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were asked for suggestions for afternoon activities in view of Scri van's 
absence; it was decided that Reinhard would continue with discussions on 
advocate teams. The group broke for lunch from 12:00 to 1:40. 

The afternoon was spent discussing various topics, including advocate 
teams, experimental design for evaluation, and criterion-referenced testing. 
About half of the audience participated actively in these discussions. At 
the conclusion of the session it was decided to reconvene at 8:30 instead 
of 9:00 the next morning, to insure ample time for Scriven's presentations. 

Thursday morning Scriven presented the value of evaluation, objectivity 
in evaluation, the Pathway Comparison Problem, and Goal-Free Evaluation. 
After lunch discussion took place on various topics, and Scriven suggested 
a checklist for product evaluation, Stufflebeam left at 2:00, due to 
difficult plane connections; Scriven and Reinhard left at 3:00. Final 
evaluation instruments were administered at 3:00, after which the Institute 
adjourned. A number of participants also left early. 

Evaluation Results 

Cognitive 

Cognitive outcomes were assessed with a Self Report of Knowledge 
(Participant Instrument #2). Tabulations of pre- and post-rasponses to 
this instrument are presented in Table 2, An analysis of variance 
performed between the two postests groups (pretested and unpretested) 
yielded no significant difference, indicating no pretest sensitization 
effect. There was also no significant difference between the pretest 
and the unpretested posttest group. However, an analysis of variance with 
repeated measures between the pretest group and the same group on the 
posttest was significant at the .05 level. It should be stressed that 
this instrument is not an achievement test; this result should be inter- 
^ Dreted only as an increase in participant perception of understand, 

ERJ.C 

ll ANOVA results are presented in Table 3. 
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Table 3 



Analyses of Variance of Scores on Participant Instrument #2, 
the Self-report of Understanding 



Pretest (0^ , n = 9) vs. same group on posttest (O2, n = 8) (Repeated Measure) 
_Sy SS df_ MS F p 



Mean 

Treatment 
S (Subjects) 
M (Measures) 
SM 



16835.06 

259.44 
175.56 
214.94 



7 
1 
7 



16835.06 

37.06 
175.56 
30,71 



5.72 



<.05 



Pretest (0^ , n = 9) vs. unpretested group on posttest (0^, n = 9) 



sv 


SS 


df 


MS 


F p 


Treatment 


53.39 


1 


53.39 


1.10 n.s 


Error 


779.56 


16 


48.72 




Total 


832.95 


17 







Pretested posttest (0^, n = 8) vs. unpretested posttest (O3, n = 9) 



SV 


SS 


df 


MS 


F p 


Treatment 


43.60 


1 


43.60 


.7649 n.s 


Error 


854.88 


15 


56.99 




Total 


898.48 


16 
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Affective 



Participant Instrument #3 was employed to assess attitudes toward evaluation 
in education; tabulations of responses to this instrument are presented in 
Table 4. Analyses of variance showed no significant difference between the 
pretest group and the unpretested posttest group, or between the two posttest 



repeated measures between the pretest group and the same group on the posttest 
was significant at the .01 level. Participants exhibited positive attitudes 
toward educational evaluation both before and after the Institute, and the 
pretested groups registered positive increases in attitude over the two days 
of the Institute. 

Critique Form 

Participant responses to the critique form were generally less positive 
than those for the Portland and Tampa Institutes. Tabulations of responses 
to items from this instrument, with means and standard deviations where 
appropriate, are presented in Tables 6 to 8. 

Pre-Insti tute Planning 

An but one of the participants indicated that they felt the topic 
of the Institute is important to educational research and practice, and that 
the topic should be treated again in future Institutes. The staff was judged 
to be very well qualified. 

Total expenses incurred by participants ranged from $90 to $600 with a 
mean of $317 and a median of $338. Personal expenses (i.e., non-reimbursable 
expense items) ranged from $0 to $338, with a mean of $42 and a median of 
$0. Only two individuals were personally responsible for expenses over $100. 
However, a majority of the participants (11) felt that the tuition fee was too 
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An analysis of variance with 



er|c ^^^9^^- 



8 









CO 








ent 




3 

s. 




1 % 




I/) 
























•r- O 




O 'r— 




* i~ 




to 




i- 3 




fO r— 








> 




O UJ 




•4—* 








CO fO 




d) c 




1/1 o 








O 4-> 








(/) o 




<U 3 




UJ 








-a 




i- 






jO 




(O 


o 


1 — 










CO 




1 a; 




+j -a 




CO 3 




O 4J 




























^ O 
















o 




c 




— c 








1 > 












Q. 




^- 




o 



o 



to 



o 

4-) ci: 



-a ro 
"a -M 

•r- 3 
ID O 



a; 

CO 



>> <D 

O rT3 
CO 

+-> -f- 



+ 


1 


1 






1 


+ 


+ 


1 






^ — ^ 






^ — ^ 
























^ ^ 










CM 








CM CM 
















CM CM 








CO 










































un 






















































^ — ^ 
























































ID KO 










































LD 






















































































































CM CM 




1— ^ 


O cr> 


1— KO 






un o> 








un 


























r— LD 








CM 








CO 


















— ' 


























































































CO CO 








CM CM 






un 




























CO 








CM 








un 












— ' 






— ' 










' 


































































m 


00 
















CO 00 


















CO 00 














CM 






































































^ 




















































































CO CO 






















t — 




,— 
















CO 










































































































00 


a\ CO 




CM CM 












00 














LD 






CM 






LT) 














































— ' 
























































































































CM CM 




CM CM 




































LT) 


CM 




CM 


















































































































1— LD 




un 


un 
















'X) Ln 


















•— ^ 




CM 


CO 
















CO 






































































































CO CO 




CM CM 


















CM CM 
























CM 


















CM 
























^■^ — " 


— " 




















1 




+ 


1 


+ 


+ 


1 


1 


+ 






CD 






































■o 










































a; 








1 






JSZ 






O 






















a; 








3 


1 




u 


c 




o 




3 


























CO 


>> fO 




3 










CO 




















c 


E 




3 


u 




E 










3 


1 




C 


CO 


o 








<o 


o 


I 






fO 3 






c 


> 


<u -o 


























c 




C 






o 


o 




i- OJ 


E O 


c 


> 








a. 










o 




o 


+J (1) 




o 


•r— 


OJ -f- 


to +-> 


CD +-) 


o 


o 


































u 
















<u . 


<u 






Ol 


3 


CO 




4-> 


-Q <D 






to 


OJ CD 


CO 3 


-M C CO 




CL 




to 


c 


^ tr. 






<u 








to 








3 


^ c: 


<D 1— 


fO O CO 


fO 


E c 








-M GO 




O 








U 




3 


fO 4-> 




r*— 




•r- 


CO fO 


•r- <U 


3 


o 




TD 


to 




^- 


c 




"O 


> 


















CO > 


c +-> u 












<u ^ 




CD to 














C 4-> 






> 


O T- 




rO fO O 


to 






3 


> 




<u 


C ^ 












> 


•r- 3 










u 


u 


> 


•r- fO 




O 


OJ 


o 


-M 






<D 




S- 






O 










O OJ 


CO 3 Q- 




u 










C 


C 




O) 


<D 


o 






CO -Q 


CO 






o +-> 




•r- "O 




CO 3 




CO 








c <u 






CO 




■o 




J > fO 


CO 


3 






CI. >, 












c 






3 






(O 










<u 


u 




c 


o »— 


C -f- 




r- <U 




c 




c c 


c 


^ Q 




•o 


<U 


c 


<u 


c 


3 CO 


u 






CO O 




O 4-* 


c 


3 




o 






o 


c= 




c 




o 




o 


CO 4-> 


o 


CO 


o 




> > 


•r- U CO 


o 


CO c 








> 










u 








<U C 






c 


O -t- +J 


•r- X 


4-> 3 -r- 






U 




o 


<U CO 


+J 


^ \A 






c 








U. Q) 


o. 




i~ 4-> 














to 


4-> 






1— 




<u 




(O 


>^ 


(O 


E 






dJ to 


.f- 3 


fO 1— u 


3 <U n3 






+J 


3 


to 


CO 


3 


O 


to 


<u 




g 


E 


u 


>^ cn 






u u 




U Ql <U 


x: 


u 


>^->-» 


O 


r~ 


u 


i— E 




-C o 


Q. 


</) 


u. 






3 




> 


<u 


C 3 


3 tf- na 


3 E 




3 


1— c 


fO 




3 


fO to 


to 


-M ^ 






o 


o 


c 




3 






O "O 


-a 3 > 


"O O -O 


> o <u 


"O 


•— <u 




> 




o 


> 


•r- U 












UJ 


to +> 


h- 




LU CO <U 


UJ u o 


UJ E 


LU 


rO E 


Q. 


UJ 






LU 


3 on 






















« 






• 


















r— 








CM 






















00 













ERIC 



to 



0) 





c 


•r— 


o 


u 










ITS 








r— 


Q. 






> 


O 


LU 


-M 








CO 




<U 


C 


CO 


o 


c 




o 




Q. 




to 


o 




13 




"a 

LU 


















II 


g 


c 










to 


1 








to 




o 




Q. 












c <: 












O 










M 






o 


c 












1 


> 


<v 









o 



to 







1 


1 






-*-> 
























to 


















CVJ CM 






o 
























Q. 






























































































o 




















^ ^ 






<D 
















































Cu 


















— 






-*-> 














- — V 










to 






CM 


CM 










WD LT) 






o 


















CO 






n 








V ^ 














CD 
























<V 
































































> 




^— ^ 




<: 


<v 














1— 1— 




CM CM 




















f— 




CM 






Q. 














■V - 


































to 


















WD !-0 






o 


















CO 






Cu 






















-a -M 
























■r- 3 








































































"a 
















































ZD o 


Q. 
















































to 






O 












CM CM 






o 






















QJ 




















„ ^ 




























CJl 
















































to 


























m 


















CM CM 




Q 




















CM 






Q. 






































- — V 










to 














CM 1— 




r— <^ 




1— (U 


o 
























Cu 














































C 
























i- CO 
























+J -r- 
























(/^ Q 


o; 








o 






00 CT> 


















o 






00 










Cu 














— ' 
















t 






1 


















0) 






o 














c 














rtJ 








1 O 


o 


£Z 












E 








3 c 




OJ o 










to 










f— 












r— 




O 


to 










4-> 








fO 












> OJ 






c 


















OJ o 




o o 


OJ 






CU 


o 




13 


•r— 








Z3 


Q- 


















f— 


> 




X 












OJ 


to 




fO ^ 




OJ OJ 


OJ 






CD 






o 


c 






















o 


o 




o o 


c 


O M- 








-o 


o 




u. 






■r- 5 . 


o 


E O 


QJ 






c 




r*— 


CL 


to 




OJ 
















J3 




■f— 




>^ F 


+J 


to +-> 


-«-> 










fO 


a; 


o 




O E 


c 


CD d 


o 












> 






3 o 


OJ 


OJ 








O M- 








o 




Q. 


3 E 










o 










OJ -r- 


to 




c 










> 


fO 


CD-M 






■r- > 


fO 








<u 




Zl 


c 




CD C <T3 


o 








Z3 


CD 






■f— 


o 


C O OJ 


OJ 


+-> 
















13 


CL 


c 










c 


OJ 


o 


> 




"O 


to +-» Ql 


o 


O E 








►—1 


r— 






E 




ZD (O 


21 


o ■.- 








o 














CVJ 
























r— 







CO 
CD 



to 
to 



OJ 



C OJ 

OJ -M 

O -r- 

OJ ^ 
Q_ O 

-M OJ 

to 
OJ 

^ o 

OJ 

C <U 

<U fO 

^ O 

-*-> on 

O OJ 



"O o 
c 

3 to 
O OJ 
r— 

o 

to 

OJ OJ 
to > 

C -r- 

o -»-> 
Q. fO 
to CD 

OJ <u 
c 

o 

> 



-M to 

c o 

OJ Q. 

o 

Cu 



o 



ERIC 



Tabls 5 



Analyses of Variance of Scores on Participant Instrument #3, 

Affective Measure 



Pretest (0^ , n = 9) vs. 


same group 


on posttest 


(O2, n 


= 8) (Repeated Measure) 


SV 


SS 


df 


MS 


F P 


Mean 


8464. 


1 


846.4 




Treatment 










S (Subjects) 


673. 


7 


96.14 




M (Measures) 


12.25 


1 


12.25 


5.12 ClO 


SM 


16.75 


7 


2.39 





Pretest (0,, n = 9) vs. unpretested group on posttest (0^, n = 9) 



SV 


SS 


df 


MS 


Treatment 


9.39 


1 


9.39 


Error 


496.89 


16 


31.06 


Total 


506.28 


17 





Pretested Posttest (0^, n = 8) vs. unpretested posttest (0^, n = 9) 



SV 


SS 


df 


MS 


F p 


Treatment 


37.77 


1 


37.77 


1.02 n.s 


Error 


553.76 


15 


36.92 




Total 


591.53 


16 







Table 6 



Tabulation of Participant Responses to Critique Items 
Relating to Pre-Insti tute Planning 



Item s 

1. How important do you feel the topic of this Institute is to 
educational research and/or evaluation? 

very important 12 3 4 5 very unimportant 1.83 1.1 

(9) (5) (3) (1) 



2. How important do you feel the topic of this Institute is 
to educational practice? 

very important 12 3 4 5 very unimportant 1.83 .86 

(7) (8) (2) (1) 



4. Leaving aside the quality of instruction for the 
moment, do you think the topic treated in this 
Institute should be treated again in future Institutes? 

definitely 12 3 4 5 definitely not 
(9) (6) CD (2) 

19. In your opinion, the staff members were in general: 

very well qualified 12 3 4 5 very 

(15) (2) (1) unqualified 



23, The meeting room facilities for the Institute were: 

excellent 12 3 4 5 poor 3.22 1.26 

(2) (3) (5) (5) (3) 



25, Considering what you have learned by attending this 

Institute (or any other benefits you have received from 
attendance), the tuition fee was: 

considerably too 1 2 3 4 5 considerably too 3.83 .79 
low (7) (7) (4) high 



1.78 1.00 



1.22 .55 



12 



Meeting room facilities were rated from excellent to poor, with the mean 
falling slightly below adequate. Several participants commented that the 
room was cramped the second day, after a partition was closed to cut down on 
noise. 

Conduct of the Institute 

Participant responses to items concerned with the conduct of the Institute 
are summarized in Table 7. Participants were asked to rate the relevance of 
the Institute content to the topic advertised; responses varied from highly 
relevant to ''not at all relevant." This reflects the highly diverse reactions 
indicated throughout the critique form; some participants responded very 
positively to the Institute, while others were very negative. 

The staff's objectives were judged to be somewhat less than clear. 
Planning and organization were rated as fair. Staff preparation and the 
quality of instruction were judged to be slightly above average, although 
responses again ranged from excellent to poor. Individual activities were 
generally rated as valuable. Eight participants felt the instruction was 
too lecture-oriented, one felt it was too discussion-oriented. Instruction 
was perceived to be interesting and informative. 

Most participants indicated that prerequisites were either ambiguous 
or not stated at all. Participant backgrounds seem to vary considerably, 
as some (5) felt they had more than enough preparation for the Institute, 
while others (2) felt they were seriously lacking in preparation. Half 
indicated that they felt too little work was required of them during the 
Institute. Advance reading materials were received in time by less than half 
of the group. 

The length of daily sessions was judged to be about right, but ten 
participants felt the two day Institute was too short to cover the topic 
adequately. 
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Table 7 

Tabulation of Participant Responses to Critique Items 
on Conduct of the Institute 



Item £ s 

3. How relevant was the content of the Institute to the topic 
that was advertised? 

highly relevant 12 3 4 5 not at all relevant 2.67 1.19 

(3) (6) (4) (4) (1) 



5. The staff's objectives for this Institute were: 

very clear to me 1 2 3 4 5 very unclear to me 3.39 1.20 

(6) (3) (5) (4) 



6. The planning and organization of this Institute was: 

excellent 12 3 4 5 poor 4.17 .92 

(6) (3) (9) 



Overall, was the Institute long enough to cover the topic 
adequately? 

considerably too 1 2 3 4 5 considerably too 3.50 .92 

long (3) (5) (8) (2) short 



As a rule, daily sessions were: 

considerably too 12 3 4 5 considerably too 3.06 .80 
long (3) (13) (2) short 



9. Do you feel all necessary prerequisites were clearly stated in information you 
received prior to the Institute? (check ONE) 

6 Y es, they were clearly stated 

6 N o, they were ambiguous 

6 N o prerequisites were listed 



10. Did you receive advance reading materials from the Institute director(s) early 
enough to read them prior to the Institute? (check ONE) 

7 Y es , materials came early enough 
. 6 N o, materials came too late 
5 No materials were sent 
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Table 7 



Tabulation of Participant Responses to Critique Items 
on Conduct of the Institute 



Item X s_ 

11. Do you think you entered the Institute with the appropriate 
prerequisites or prior knowledge to make what you learned 
there of value to you? 

I had more than 1 2 3 4 5 I was seriously 2.56 1.34 

enough preparation (5) (4) (5) (2) (2) lacking in 

preparation 

12. In terms of your background and preparation for the Institute, 
the content of this Institute was: 

too 1 2 3 4 5 too 2.94 .80 

elementary (1) (3) (10) (4) advanced 



13. Overall, the quality of instruction in this Institute was: 

excellent 12 3 4 5 poor 2.67 1.24 

(3) (7) (2) (5) (1) 



14. The instruction was generally: 



too lecture- 
oriented 


1 

(3) 


2 

(5) 


3 

(9) 


4 


5 

(1) 


too discussion- 
oriented 


2. 


,50 


very 

interesting 


1 

(5) 


2 

(5) 


3 

(6) 


4 

(2) 


5 


very uninteresting 


2. 


,28 


very 

informati ve 


1 

(6) 


2 

(3) 


3 

(6) 


4 

(3) 


5 


very uninformati ve 


2, 


.33 



15. Opportunities for asking questions were: 



sufficient 1 2 3 4 5 insufficient 
(7) (5) (2) (3) (1) 



2.22 1.31 



16. Opportunities for studying were: 

sufficient 12 3 4 5 
(3) (5) (6) (3) (1) 



insufficient 



2.67 1.14 
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17. The amount of work the staff required of you was 
far too much 1 2 



3 4 5 far too little 
(9) (8) (1) 



3.56 .62 
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Table 7 

Tabulation of Participant Responses to Critique Items 
on Conduct of the Institute 



Item x_ s_ 

18. Opportunities for you to interact with the staff with respect 
to problems in your own work which relate to the topic were: 

sufficient 1 2 3 4 5 insufficient 3.12 1.36 

(2) (4) (5) (2) (4) 



19. In your opinion, the staff members were in general: 

very well- 12 3 4 5 very unprepared 2.44 1.50 

prepared (6) (6) (1) (2) (3) 



20. Did the staff seek your reactions to their instructional 
procedures, scheduling, etc., during the Institute? 

frequently 1 2 3 4 5 never 3.00 1.46 

(3) (5) (3) (3) (4) 



21. Did it appear to you that your reactions led to improvements 
in the instructional procedures, scheduling, etc.? 

frequently 1 2 3 4 5 never 3.35 1.17 

(1) (2) (8) (2) (4) 



22. The formal evaluation of this Institute by the "outside evaluation" 
(e.g., those instruments) was: 

not at all 12 3 4 5 very disruptive 1.71 1.05 

disruptive (11) (1) (4) (1) 
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Table 8 

Tabulation of Partiwipant Responses to Critique Items 
on the Evaluation of Institute Activities 



30. How would you rate the value of each of the following aspects of this Institute? 
(Please circle ONE for EACH row) 



no opportunity very 
to judge valuable worthless 



a. Scriven: pathway evaluation NA 



1 2 3 4 5 
(6) (6) (5) (1) 



2.06 .94 



b. Stufflebeam: problems, NA 
CIPP model 



1 2 3 4 5 
(5) (6) (4) (1) 



2.05 .93 



c. Reinhard: advocate team NA 
technique 



1 2 3 4 5 
(3) (4) (5) (2) (2) 



2.75 1.29 



d. Scriven: goal-free 
evaluation 



NA 



1 2 3 4 5 
(10) (4) (1) (2) 



1.71 1.05 



e. Workbook examples 



- CI ; 



NA 



1 2 3 4 5 
(2) (3) 



2.60 .55 



f. Individual reading: use of NA 
reference table 



1 2 3 4 5 
(3) (4) (2) (1) 



3.10 .99 



g. "Special topics" presentation NA 



1 2 3 4 5 
(3) (4) (1) (1) 



3.00 1.00 



h. Discussion of individual NA 
problems 



1 2 3 4 5 
(3) (3) (5) (3) 



3.57 1.09 



i. General question and answer NA 
sessions wi th staff 



1 2 3 4 5 
(2) (5) (5) (1) (2) 



2.73 1.22 



j. Informal discussion with peers NA 



1 2 3 4 5 
(3) (6) (1) (4) (1) 



2.60 1.30 
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Seven participants indicated that they were seldom asked for their 
reactions or suggestions during the Institute. However, Stufflebeam asked 
for participant feedback several times, and very little was offered. Negative 
responses to this item may reflect generally negative attitudes toward the 
Institute, rather than actual perceptions of the staff's flexibility. 

Open-ended Questions 

Responses to the open-ended questions also varied from very positive to 
very negative. Suggestions for improving the Institute are summarized in 
Table 9. The most recurrent complaint (6) concerned Scriven's late arrival. 
Aspects of the Institute judged to be of considerable value are listed in 
Table 10. 

General Evaluation of the Institute 

Participant responses relating to a general evaluation of the Institute 
are presented in Table 8. A majority of the participants (13) believed that 
vvhat they had learned would be useful to them in their own work; three felt 
that their learning experiences would not be useful to them. Slightly over 
half of the respondents indicated that they would choose again to attend 
the Institute (10) and would recommend attendance to a colleague (11). 

Summary and Recommendations 

Although participant reactions to this Institute are less positive than 
for the Portland and Tampa Institutes, mean responses are only occasionally 
negative. Responses to most items varied across the scale. It is possible 
that the more negative respondents were reacting primarily to Scriven's absence 
the first day and allowed this to affect their responses to all items^ while 
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the more positive respondents evaluated the Institute itself. It is the 
opinion of this evaluator that the Tucson Institute was better planned and 
conducted than the one in Portland, and that the series should definitely 
continue. However, there are a few suggestions for the improvement of the 
final presentation in July. 

1. All staff members should arrive on time and remain until the end of 
the two days. Although it is realized that Scriven's late arrival was not 
deliberate and Stufflebeam's early departure was unavoidable, several 
participants expressed resentment of these circumstances, both at the Institute 
and through instrument responses. This should be avoided in the future if at 
all possible. 

2. Clearly stated objectives and a detailed agenda should be mailed to 
participants in advance of the Institute, to give them a better idea of what 
emphases to expect. 

3. The billing of the Institute should be changed, or applicants should 
be notified that "alternative conceptions" refers primarily to Scriven, 
Stufflebeam, and Reinhard's conceptions of evaluation^ which complement each 
other more than they differ. 

4. Pre-Insti tute materials should be mailed first class at an earlier 
date, to insure their receipt by more participants. 

5. More work should be expected of participants during the Institute, 
and a greater emphasis should be put on the value and availability of the 
resource materials . 

6. More emphasis should be placed on discussion rather than lecture. 



Table 9 



Participant Suggestions for Improving the Institute 
from Responses to Open-Ended Questions 



Staff should arrive on time and stay until the end of the Institute. 

Division into small groups. 

Less philosophy and more application. 

Hold Institute on Thursday and Friday. 

Revise advertised content and include agenda. 

Mail advance materials earlier. 

More emphasis on alternative concepts of evaluation. 
Discussion of the administrative viewpoint. 
Less lecture, more discussion. 
Better preparation. 
Smaller frame of reference. 



Table 10 

Aspects of the Institute Believed to be of Considerable Value 



Good bibliography. 

Materials sent prior to the Institute. 

Exposure to new vocabulary, ideas, and techniques. 

Contact with colleagues. 

Personal interaction with leaders. 

Discussion of philosophical differences. 



EVALUATION OF AERA TRAVELING INSTITUTE NO 
"Alternative Conceptions of Evaluation 
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Thir. reuort contains an evaluation of the second AERA Traveling 
Institute, the "Alternative Conceptions of Evaluation" Institute con- 
ducted by Michael Scriven and Daniel Stuff! ebeam in Portland, Oregon, 
on October 19 and 20, 1972. There are two primary audiences for the 
report: (1) the AERA Research Training Committee, which will use it 
as input for decisions about continuation of institutes in this series, 
and (2) the Institute staff (Scriven, Stufflebeam, and Reinhard), who 
should find it useful in deciding how to improve the Institute in its 
second run (assuming the Research Training Committee agrees with the 
recommendation of the evaluators that the Institute should be repeated). 

Other audiences which might have an interest in the report include 
the Office of Education, funding agency for the project; participants 
of the Institute, several of whom expressed interest in the report in 
Portland; and prospective participants who are considering attending 
subsequent institutes in this series. To the extent that it is feasible, 
requested report copies should be made available to these additional 
audiences • 

A summative evaluation of the concept of Traveling Institutes is 
planned to help AERA determine whether they should continue to sponsor 
such institutes beyond the current experimental series. However, this 
evaluation will be deferred until all twelve institutes (4 series of 
3 institutes each) are completed; therefore, no conclusive inferences 
can be made about the concept of AERA Traveling Institutes on the basis 
of the evaluation presented herein. 

The remainder of this report is divided into two major sections, 
description of the Institute and evaluation of the Institute. The 
descriptive section is sub-divided into four sections which contain 



descriptions of the follovnng: (1) policies under which the evaluation 
was conducted, (2) planning of r.hn Institute, (3) Institute participants, 
and (4) the conduct of the Institute. 

The evaluation of the Institute is also divided into four sections 
which contain discussions of the following: (1) evaluation plans and 
procedures, (2) instrument administration and analysis, (3) results, 
and (4) sumnary and recommendations. 



DESCRIPTION OF THE INSTITU TE 
The Institute is described briefly in the following four sections. 

P olicies Under Which the Evaluation Was Conducted 

The evaluators were given complete access to all relevant infor- 
mation in the possession of the AERA Central Office staff and the 
Institute staff. This included participant application forms and records 
of planning for the Traveling Institutes. The Institute staff cooperated 
by supplying a preliminary agenda, with the understanding that it was sub- 
ject to modification during the course of the Institute. A copy of this 
agenda appears in Appendix A herein. The directors also agreed to provide 
ample time for the administration of all evaluation instruments during 
the Institute. 

The evaluation budget for this Institute was sufficient only to 
cover expenses associated with the evaluation (e.g., travel costs, 
living expenses, reproduction costs for instruments and reports). The 
evaluators' time was volunteered , resulting in some limitations in terms 
of (1) the amount of time which could be devoted to the task and (2) schedul- 
ing of that time. As a result, the analyses included and the written inter- 
pretations of analyses are less complete than would have been the case had 
it been possible to spend more time on these tasks J Another result is 
that this report is somewhat late for maximum utility for decisions for 
which it is relevant. These limitations notwithstanding, it is hoped 
that this report proves useful to the audiences described earlier. 

Vor example, stratifications on several additional variables would 
have been useful, as would also pilot testing and item analyses of 
instruments. 
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Institute Planning 

Planning and objectives for t.his Institute v/ere made available to 
the evaluators through telephone conversations and personal co^^respon- 
dence with (a) Scriven and Stuff lebearn, the Institute directors; 
(b) James Mitchell, Model A director; and (c) William Russell, AERA 
central office project co-directcr. 

Stufflebeam stated four basic goals for the Institute: 

(1) to increase participant awareness and understanding of 
problems with which evaluators deal; 

{?.) to provide an awareness of and greater knowledge about 

alternative frameworks for dealing with evaluation problems; 

(3) to provide information on emergent techniques for working 
on evaluation problems (e.g., the advocacy team approach); 
and 

(4) to increase participant abilities to solve evaluation 
problems. 

Sc^iven*s objectives, as described in a telephone conversation 
with the evaluators, were more general, but did not differ significantly 
from Stuff lebeam's , and Reinhard was not consulted concerning objectives 
prior to the Institute. 

Institute Participants 

Participan+5 were chosen on the basis of their expressed professional 
need for the content of the Institute, as indicated in a personal statement 
submitted with each application form. (A copy of the application form 
is included in Appendix B.) Out of 37 applications submitted, 26 
participants were selected, 23 of which attended the Institute in Portland. 



In addition, one person attended as a replacement for a person who had 
been selected but who was unable to attend. A list of participants is 
presented in Appendix C. 

Table 1 contains a description of Institute participants on 
several characteristics. Institutions of higher education and public 
school systems collectively employ approximately 80 percent of the 
participants. Almost half of the participants are employed as evalu- 
ators , with teachers and professors representing the next largest 
group. Nearly all participants hold graduate degrees, with a pre- 
ponderance holding the doctorate. All but 2 participants are members 
of AERA; they show considerable diversity in their divisional 
membershi ps • 

All participants came to the Institute with at least limited 
experience in the field, and over half indicated that they had a consider- 
able or great amount of experience. No predominate theme could be 
discerned in participants' reasons for attending the Institute, 

Table 1 

Description of Participants 

Employing Institution 

College or Public School Regional Federal or State Dept. 
University System Laboratory City Govt of Education Other 

12 7 1 1 12 



Primary Professional Role 
Teacher Administrator Researcher Evaluator 

7 4 1 11 



Developer 
1 



Table 1 (conti nuodj 
Description of Participants 



Bachelor ' s 
1 



Academic Degrees 
Master ' s 
4 



Doctorate 
19 



Membership in Divisions of AERA 



(A) Administration (B) Curriculum & 

Objectives 



1 



(C) Learning & (D) Measurement & 
Instruction Research 

Methodology 

9 7 



( ) Social Context 
of Education 



1 



( ) School Evaluation 
& Program 
Development 

7 



( ) Not presently 
a member of 
AERA 



Reasons for Attending the Institu te 



Desire to gain new 
skills or knowledge 
related to the topic. 



19 



Desire to improve 
existing skills or 
knowledge related 
to the topic. 

18 



Desire to discuss 
problems related 
to the topic 
with experts 

20 



Prior Experience 



No 

Experience 
0 


Limited 
Experience 

8 




Considerable 
Experience 

11 


Great Amount 
of Experience 

5 




Distance of 


home from 


Institute site 




less than 


50-100 


101-500 


501-1,000 


more than 


50 miles 


miles 


mi les 


miles 


1 ,000 miles 


4 


0 


4 


4 


12 
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Perhaps tlie most surprising fact about this group of participants 
is that two-thirds of them travelled over 500 miles (one-way) to attend 
the Institute; half traveled over 1,000 miles (one-way). This suggests 
that the immediacy of professional needs or interest attracted more 
participants than did the regional convenience. 

Conduct of the Institute 

"Pre-test" evaluation instruments were administered to participants 
as they arrived at registration, prior to the opening of the Institute. 
Following this, all participants were encouraged to use the remainder of 
the morning for individual study. No materials had been distributed in 
advance of the Institute, but individual material sets were provided upon 
arrival for each attendee. In addition, a resource area was set up 
offering an assortment of materials related to evaluation. A bibliography 
and matrix describing these resource materials appears in Appendix D. 
Most of the morning of October 19th (9:00 - 11:00) had been planned 
initially as a reading session. This was extended to 1:00 p.m. because 
of Scriven's delayed arrival, which was due to bad weather and attendant 
plane delays. 

Presentations began at 1:00 p.m. with an introduction and Scriven's 
address on the "Pathway" model of evaluation. This was followed by Stuffle- 
beam speaking on problems faced by evaluators, and Reinhard with an over- 
view of the advocacy team approach to evaluation. Following a short break 
at 3:30, the group reconvened to hear Scriven discuss goal-free evaluation, 
after which Stufflebeam spoke on on-site evaluation procedures. Instruc- 
tions were then given for the evening's activities; participants were 



divided into groups and asked to devise evaluative strategic: and 

critiques in response to problenis proposed in +he Sanders and McClellan 

p 

(1972) evaluation workbook materials. Following dinner, groups v;ere 
asked to report their strategies, and these were critiqued by other 
participants. At the conclusion of this exerc.se, input was requested 
from participants concerning plans for activities on the second day. 

The tentative agenda was followed less rigorously on the second 
day. The morning was spent discjssing evaluation problems posed by 
participants, followed by Stuff lebeam speaking on school system evaluation 
Following lunch, Scriven gave a presentation on behavioral objectives and 
needs assessment. Some time was spent on general questions and discussion 
At 3:00 p.m., final evaluation instruments were administered, after 
which the Institute adjourned. 



Sanders, James R. and McClellan, Mary C. "A Workbook for the 
study of Applications of Selected Evaluation Frameworks/' Indiana 
University, 1972. 



EVALUAT ION OF THE INSTITUTE 

The methods used to evaluate the Institute and the results of that 
evaluation are contained in the remainder of this report. 

Evaluation Plans and Procedures 

The plans and objectives for the experimental series of AERA 
Traveling Institutes v/ere studied as a first step in planning the 
evaluation of this Institute. This led to a second step, listing 
evaluative questions for which ansv/ers must be provided in order to 
judge the worth of each Institute. Third, sources of information neces- 
sary to answer each question were identified and categorized. These 
categories and relevant types of information are summarized briefly below 

(1) from participants -- personalogical data, information on cognitive 
and affective changes (both short and long term) resulting from the 
Institute, and personal evaluations of various aspects of the Institute; 

(2) from staff members reactions to the Institute; and (3) from 
non-attendees (including non-applicants and those who were accepted but 
did not attend) information concerning their interest or non-interest 
in attending the Institute. An outline of information needs is included 
in Appendix E. 

With these general information needs in mind, instrument formats 
were planned. Instruments to be administered on site were then drafted, 
reviewed, and revised. Copies of all instruments are included in 
Appendix F. 

It was difficult to assess cognitive outcomes of this Institute, 
due to uncertainty as to precisely what topics would be covered during 
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the Institute. Consequently, no cognitive achievem'int test v/as admini- 
stered; cognitive gains were estimated by a self-report of understanding 
(Participant Instrument #2)/ 

Participant Instrument #3 was devised to assess affective outcomes 
of the Institute. It consists of 12 statements relating to the use of 
evaluation in education; participants v/ere asked to respond to these on 
a 5-point Likert type scale. 

General evaluative data from both staff and participants were gathered 

from two critique forms (Staff Instrument #1 and Participant Instrument #4) 

4 

consisting primarily of semantic differential and open-ended questions. 

Demographic information was collected from a short instrument 
(Participant Instrument #1) concerned primarily with participants' 
background in evaluation anu current professional roles. 

Instruments for follow-up on participants, non-attendees, and 
non-applicants have not been completed, but will be finalized and 
administered to the appropriate groups in approximately four months 
(April , 1973). 



The evaluators hope to develop a more objective measure of cog- 
nitive outcomes for Institute Two in order to assess the reality of 
participants' perceptions of their understanding of evaluation. 

4 

Other information concerning participant and staff reactions to 
the Institute was obtained from talking informally with both groups 
during the Institute, and from subsequent correspondence between parti- 
cipants and staff which was supplied to the evaluators by members of 
the staff (Appendix G). 



Instrument A clmiMistr a tion and Analyses 

Instrument f-l was administered to each participant prior to the 

beginning of Institute activities. At the same time, eleven of the 

attendees were randomly selected to respond to Participant Instruments 

§2 and if3 (the cognitive and affective measures); these instruments were 

also given to the entire group as a post-test at the conclusion of the 

5 

Institute. The resulting evaluation design appears below. 

R 0^ X O2 
R O3 

Participant Instrument #4 and Staff Instrument were also admini- 
stered at the conclusion of the Institute to all persons in their respec- 
tive groups. 

The following analyses were performed on the data: 

(1) descriptive statistics, comprising simple tabulations with 
means and standard deviations where appropriate, for all 
instruments . 

(2) one way ANOVAs between total pre-and post-test scores on 
Participant Instruments ifZ and #3, for the following compari- 
sons: 0^ - 0^ - 0^, and O2 - O^. 

(3) contingency tables; employing institution and total amount of 
money expended for the Institute by each participant were used 
as stratifying variables and plotted against responses to 
items 4, 23 and 24 on Participant Instrument #4. 

5 

Subscripts are used to facilitate the identification of groups in 
the description or analyses performed. 



An evaluative procedures for this Institute were conducted by the 

presen:: authors, with assistance on computer analyses from Roy Gabriel 

and Don Phi 1 1 i ps graduate fellows of the Laboratory of Educational Poseorch, 
University of Colorado. 



Evaluation Resu lts 

The results of this evaluation are organized under the three basic 
evaluative instruments which were used in the evaluation. 

Self-Report of Knowledge 

Tabulations of pre- and post-test responses on the self rpnort of 
knowledge are presented in Table 2. Analyses of variance were performed 
between total pre- and post-test scores on this instrument; results are 
presented in Table 3. Significant differences were found at the .05 
level both between the pretest (0^) and the same group on the post-test 
(Op) y snd between the pretest (0^) and the unpretested post-test group 
(O^). Apparently, participant^perceived tt^emselves as havina signifi- 
cantly increaspri their understanding or evaluation-related topics. 
However, it is interesting to note that participants indicated consider- 
able gains on item 6, "evaluation desiyns, including time series vs. 
control groups," a topic which was not treated in this Institute. It is 
possible that the resource table operated to increase actual participant 
knowledge relating to evaluation design, in which case these perceived 
gains in knowledge could reflect real gains. However, it seems equally 
possible that these perceptions are unrealistic and merely reflect an 
increase in attitude toward the subject; perhaps this instrument assesses 
affective rather than cognitive outcomes. In short, Instrument ^3 did not 
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Table 3 



Analyses of Variance of Scof.'S on Participant Instrument r2, 
the Self- report or Knowledge Inventory 



Pretest (0-,, n = 11) vs. sane group on post-test (Op, n = 11) 



SV SS df MS F p 

Treatment 290.91 1 290.91 10.23 < .01 

Error 568.55 20 28.43 
Total 859.46 21 



Pretest (0^, n = 11) vs. unpretested group on post-test (O^, n = 13) 

SV SS df MS F p 

Treatment 702.74 1 702.74 27.85 < ,001 

Error 555.22 22 25.24 
Total 1257.96 23 



Pretested post-test (O^, n=11) vs. unpretested post-test (O^, n = 13) 

SV SS df MS F p 

Treatment 76.68 1 76.68 3.11 n.s. 

Error 541.94 22 24,63 

Total 618.62 23 



supply important data about whether participants actual 1y increased their 
knowledge in evaluation topics covered in the Institute. It did show that 
participants felt they increased their knowledge. 



No significant differences were found between the two post-tect 
groups, suggesting that there was no pretest sensitization effect for 
this instrument. 

Attitu de s to wa rd Ed uc ational Ev f luation 

Tabulations of responses to Participant Instrument fi3, the affec- 
tive measure, are presented in Table 4. An analysis of variance per- 
formed on these results (Table 5) showed a significant difference 
between the pretest (0^ , x = 26.5, S = 4.5) and the unpretested post- 
test group. ^0^, X = 22.5, S = 3.5) at the .05 level, but no significant 
difference in a repeated measures ANOVA between the pretest (0^) and the 
same group on the post-test (O2, x = 27.0, S = 4.7).^ The two post-test 
groups (O2 and 0^) were also found to differ significantly at the .05 
level. These results are puzzling, for it seems that while the pretest 
did not actually desensitize people to the treatment (i.e., those who 
took the pretest did not score less positively on the retest), it may 
have functioned as a retardant to affective growth. Since two pretests 
(cognitive and affective) were given to the same participants, it would 
be difficult to infer any kind of causal relationship, but there does 
seem to be some sort of differential effect in pos >test responses of 
the pretested group as compared with the unpretestei group. 



Responses were scored with ''V denoting the most positive 
responses and "5'' denoting the most negative responses. 
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Table 5 



Analysis of Variance of Scores on Participant Instrument ^3, 
Inventory of Attitudes toward Educational Evaluation 



Pretest (0,, n = 11) vs. same group on post-test (0^, n = 11) 



Sy SS df ms_ F p_ 

Treatment 1.64 1 1.64 .08 n.s. 

Error 424.73 20 21.24 

Total 426.37 21 



Pretest (0;j , n = 11) vs. unpretested group on post-test (O^, n = 13) 

S V SS df ms F £ 

Treatment 95.00 1 95.00 6.04 < .05 

Error 345.96 22 15.73 

Total 440.96 23 



Pretested post-test (O2, n = 11) vs. unpretested post-test (O^, n = 13) 

SV SS df ms F p 

Treatment 122.73 1 122.73 7.23 < .05 

Error 393.23 22 16.97 

Total 495.96 23 
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Cri t i_q^u o Form 

Participant reactions to the Institute, especially as surveyed by 
the critique form (Participant Instrument M), tended to be very positive, 
with a few aspects of the Institute singled out for criticism. Tabulation 
of responses to most items from this instrument, along with means or 
standard deviations where appropriate, are presented in Tables 6 to 13. 
Some items have been deleted from this report, as they pertain to 
evaluation of the concept of Traveling Institutes rather than to the 
evaluation of this particular Institute. The remaining critique form 
items have been grouped into four logical clusters, and the results in 
this section have been subdivided into four corresponding sections. 

Evaluation of pre-Insti tute planning . Participants were asked to 
record their impressions of pre-Insti tute planning in areas related to 
the topic, staff, facilities, fee structures, location, and date. 
Participant responses to these questions are summarized in Table 6. 

In general, participants responded very positively to the importance 
of the Institute topic, both in relation to educational research and 
educational practice. All but one of the participants felt that the 
topic should be treated again in future Institutes. The staff selected 
to present the topic was judged to be very well-qual ified. 

Participants were asked to record totjj^ expenses (including travel, 
tuition fee, living expenses, etc.) they had incurred by attending the 
Institute, as well as the amount of money they personal 1y had to spend 
to attend the Institute (i.e,, non-reimbursable expense items). Total 
costs ranged from $100 to $700, with a mean of about $320 and a median 
of $300. However, personal expenses ranged only from $0 to $350 with a 
mean of about $30 and a median of $10. Only one individual was personally 
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Table 6 

Tabulation of Participant Responses to Critique Items 
Relating to Pre-Institute Planning 



Item X s 

1. How important do you feel the topic of this Institute is to 
educational research and/or evaluation? 

very important 12 3 4 5 very unimportant 1.46 .93 
(17) (5) (1) (1) 

2. How important do you feel the topic of this Institute is to 
educational practice? 

very important 12 3 4 5 very unimportant 1.63 .97 

(14) (7) (2) (1) 

4. Leaving aside the quality of instruction for the moment, do 
you think the topic treated in this Institute should be 
treated again in future Institutes? 

definitely 12 3 4 5 definitely not 1.30 .55 

(18) (5) (1) 

18. In your opinion, the staff members were in general: 

very wel 1-qual i fied 12 3 4 5 very 1.21 .41 

(19) (5) unqualified 

22. The meeting room facilities for the Institute were: 

excellent 12 3 4 5 poor 2.45 1.18 

(5) (9) (5) (2) (2) 

24. Considering what you have learned by attending this 
Institute (or any other benefits you have received 
from attendance) 5 the tuition fee was: 

considerably too low 12 3 4 5 considerably 3.45 1.02 

(16) (5) (3) ^''9^ 



ERLC 



Note: Frequencies are shown in parentheses, 
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responsible for expenses over $100, altfiough one other did not yet know 
how much would be reimbursed to him. 

Eight of the 24 participants felt that the tuition fee was too high, 
whereas Ic participants responded neutrally to the question. However, it 
should be remembered that in only two cases were participants themselves 
responsible for the tuition fee. Stratification of responses according to 
the total amount of money spent for each participant to attend the Institute 
(Tables 7, 8, and 9) suggests that there is no relationship between amount 
of money paid'and attitude toward the amount of the tuition fee, the topic, 
or usefulness of the Institute. 

The date and location of the Institute seem to have been convenient 
for most participants (data from Participant Instrument #1); only five felt 
the site was inconvenient, even though twelve participants traveled over 
1,000 miles to the Institute. This suggests that the immediacy of profes- 
sional needs or interests attracted more participants than did the regional 
convenience. 

"eeting rooms were rated from excellent to poor. A few participants 
mentioned that noises from the street and other meeting rooms were sometimes 
distracting. However, most participants felt the meeting rooms ranged from 
adequate to good. 

Evaluation of conduct of the Institute . Participants were also asked 
to respond to questions dealing with the conduct of the Institute itself, 
including concerns for the success of its activities, content validity, 
objectives, and instruction. Summaries of responses to these questions are 
presented in Table 10. 
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^ Table 7 

Summary of Responses cn Item 4, Stratified by 
Total Amount Spent for Each Participant to Attend Institute 



Item 4: Leaving aside the quality of instruction for the moment, do you 

think the topic treated in this Institute should be treated again 
in future institutes? 



Total $ Spent to 
Attend Institute 


iDefinitely 




Def ini tely 
Not 


Total 




1 


2 


3 


4 5 




$ 0 to $150 


4 

(100) 








4 

(100) 


$151 to $250 


4 

(80) 


1 

(20) 






5 

000) 


$251 to $350 


2 

(33) 


3 

(50) 


1 

(17) 




6 

(100) 


$351 to $450 


5 

(100) 








5 

(100) 


More than $450 


3 

(75) 


(25) 






4 

(100) 


Total 


18 
(75) 


5 

(21) 


1 

(4) 




24 
(100) 



Note: Percentages of row total^ are shown in parentheses below each 
frequency. 
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Table 8 



Summary of Responses on Item 23, Stratified by 
Total Amount Spent for Each Participant to Attend Institute 



Item 23: Do you anticipate that what you have learned from the Institute 
will be useful to you in your own work? 



iotal $ Spent to 
Attend Institute 


Definitely 
1 2 


3 


Definite! y 

Not 
4 5 


Total 


$ 0 to $150 


2 


1 


1 

(25) 




4 

(100) 


%151 to $250 


(60) 


2 

(40) 






5 

(100) 


$251 to $350 


2 

(33) 


1 

(17) 


2 

(33) 


1 

(17) 


6 

(100) 


$351 to $450 


3 

(60) 


1 

(20) 


1 

(20) 




5 

(100) 


More than $450 


1 

(25) 


1 

(25) 


1 

(25) 


1 

(25) 


4 

(100) 


Total 


11 

(46) 


6 

(25) 


5 

(21) 


2 0 

(S) 


24 

(100) 



Note: Percentages of row totals are shown in parentheses below 
each frequency. 



Table 9 



Summary of Responses on Item 24, Stratified by 
Total Amount Spent for Each Participant to Attend Institute 



Item 24: Considering what you have learned by attending this Institute 
(or any other benefits you have received from attendance), 
the tui tion fee was : 



Totdl $ spent to 
Attend Institute 


Considerably 
too low 




Considerably 
too high 


Total • 




1 2 


3 


4 


5 




$ 0 to $150 




3 

(75) 


1 




4 

(100) 


$151 to $250 




4 

(80) 


1 

(20) 




5 

(100) 


$251 to $350 




4 

(57) 


1 

(17) 


(16) 


6 

(100) 


$351 to $450 




2 

(40) 


(20) 


2 

(40) 


5 

(100) 


More than $450 




3 

(75) 


1 

(25) 




4 

(100) 


Total 


0 0 


16 

(67) 


5 

(21) 


3 

(12) 


24 
(100) 



Note: Percentages of row totals are shown in parentheses below each 
frequency. 
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Table 10 

Tabulation of Parti cipi-nt Responses to Items 
on Conduct of the Institute 



Item X s 

3. How relevant was the content of the Institute to the 

topic that was advertised? 2.08 .93 

highly relevant 12 3 4 5 not at all 

(6) (12) (5) (1) relevant 

5. The staff's objectives for this Institute were: 3.00 .98 

very clear to 1 2 3 4 5 very unclear 
me (9) (8) (5) (2) to me 

6. The planning and organization of this Institute was: 2.79 1.10 

excellent 12 3 4 5 poor 
(3) (6) (10) (3) (2) 

7. Overall , was the Institute long enough to cover the -j 96 
topic adequately? 

considerably 12 3 4 5 considerably 
too long (1) (!) (15) (3) (4) too short 

8. As a rule, daily sessions were: 2.65 .57 

considerably 1 2 3 4 5 considerably 
too long (1) (6) (16) too short 

9* Do you feel all necessary prerequisites were clearly stated in 
information you received prior to the Institute? (check ONE) 

7 Yes, they were clearly stated 
10 No> they were ambiguous 
7 No prerequisites were listed 

10. Did you receive advance reading materials from the Institute 

director(s) early enough to read them prior to the Institute? (Check CUE) 



Yes, materials came early enough 



Q 1 No, materials came too late 

ERIC 22 No materials were sent 
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Table 10 

Tabulation of Participant Responses to Items 
on Conduct of the Institute 



Item X s 

11. Do you think you entered the Institute with the approp- 
riate prerequisites or prior knowledge to make what you 

learned there of value to you? 2.0 .88 

I had more than 12 3 4 5 1 was seriously 
enough preparation (8) (9) (6) (1) lacking in preparation 

12. How often did the instructional procedures take into 
account variability in prior knowledge brought to the 

Institute by participants? 3.0 .95 

always "i 2 3 4 5 never 
(1) (7) (6) (9) 

13. Overall, the quality of instruction in this Institute 

was: 2.25 .90 

excellent 1 2 3 4 5 poor 
(4) (13) (4) (3) 

14. The instruction was generally: 

too discus- 

too lecture-oriented 12 3 4 5 sion-oriented 3.04 .55 

(3) (17) (4) 

very interesting 12 3 4 5 very uninteresting 2.08 .78 

(6) (10) (8) 

very informative 1 2 3 4 5 very uninforniative 2.13 .85 

(6) (10) (7) (1) 

15. Opportunities for asking questions and studying were: 

sufficient 12 3 4 5 insufficient 1.79 .93 

(10) (11) (2) (1) 

16. The amount of work the staff required of you was: 3.50 .72 

far too much 1 2 3 4 5 far too little 

(15) (6) (3) 
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Table 10 



Tabulation of Participant Responses to Items 
on Conduct of the Institute 



Item X s 

17. Opportunities for you to interact with the staff with 
respect to problems in your own work which relate to 

the topic were: 3.25 1.1 

sufficient 1 2 3 4 5 insufficient 
(3) (2) (7) (10) (2) 

18. In your opinion, the staff members were in general: 2.13 1.08 

very wel 1 -prepared 12 3 4 5 very 

(7) (12) (5) unprepared 



19. Did the staff seek your reactions to their instructional 

procedures, scheduling, etc. during the Institute? 1.58 .83 

frequently 12 3 4 5 never 
(15) (4) (5) 



20. Did it appear to you that your reactions led to improve- 
ments in the instructional procedures, schedules, etc? 2.46 .83 

frequently 12 3 4 5 never 
(4) (8) (9) (3) 



21. The formal evaluation of this Institute by the "outside 

evaluation'* (e.g., those instruments) was: 1.29 ,62 

not at all disruptive 1 2 3 4 5 very dis- 

(17) (3) (2) ruptive 



Note: Frequencies are shown in parentheses. 



Participants tended to view the content of the Institute as relevant 
to the topic advertised. However, several (7) participants believed that 
the staff's objectives were unclearly stated. A majority of participants 
indicated that prerequisites for the Institute v/ere stated ambiguously or 
not at all. Apparently this did not result in participants "getting in 
over their heads," since all but one of the participants said they were 
at least adequately prepared, and most believed they had more than enough 
preparation. This last item may signal a problem in the reverse direction; 
some participants may have felt they were too prepared for the content of 
this Institute. Several participants indicated that the instructional 
process seldom considered individual differences in preparation for the 
Institute. 

In terms of scheduling, the weight of opinion leaned to the view that 
the Institute was too short to cover the topic adequately, whereas the 
daily sessions tended to be viewed as too long. 

The quality of instruction was rated as generally good; it was viewed 
as interesting, informative, and seemed to have achieved the right balance 
between lecture and discussion. General planning and organization for the 
Institute received varied ratings, with about 35 percent of the participants 
rating them positive, 20 percent negative, about half neutral. Most partici- 
pants felt the staff members were well prepared, although five believed them 
to be somewhat unprepared. This observation was amplified by informal remarks 
from several participants who indicated that although the presentations 
were for the most part consistently interesting, it seemed at times that 
staff members were "talking off the tops of their heads" about general con- 
cerns in evaluation, with little attention to the integration of interesting 
bits and pieces within the context of this Institute. These comments. 



although offered by a minority of the participants v/ere consistent v/ith 
the evaluators' perceptions that the presentations were interesting and 
provocative, but were not integrated into a cohesive framework to aid 
participants integrate the various concepts. 

Interestingly, a majority of the participants felt that the Institute 
did not demand enough work on their part. 

Most participants felt they had ample opportunities to provide feed- 
back to the staff on scheduling, instructional procedures, etc, and a 
majority felt that feedback led to improvements in those areas. 

Several participants stated early in the Institute that they v/ere 
attending primarily because of the reputations of the directors (one even 
mentioned something about obtaining autographs, but the evaluators failed 
to follow up to see if this objective had been attained). Some partici- 
pants were interested primarily in hearing Scriven and Stufflebeam in 
person, while others clearly hoped to corner the staff at some point for 
consultation on personal problems and seemed to feel such opportunities 
would be provided, as indicated in the Educational Researcher description 
of the Institute (Appendix H). When questioned about opportunities for 
interaction with the staff concerning personal problems relating to the 
topic, half of the participants stated that they felt these opportunities 
were somewhat insufficient. 

Evaluation of Institute activities . Item 29 asked participants to 
evaluate specific activities during the Institute; a summary of their 
responses to this question is presented in Table 11. 

Some of the most negative criticisms of the Institute were directed 
toward the Thursday morning reading session. Although ratings ranged 
from "very valuable" to "worthless," many participants suggested, both 
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Table 11 

Tabulation of Participant Responses to Items on the 
Evaluation of Institute Activities 



29. How would you rate the value of each of the following aspects of 
this Institute? (Please circle ONE for EACH row) 

no opportunity very 

to judge valuable worthless 

a* Scriven on accreditation NA 1 



b. Scriven's initial presen- 
tation (Pathv/ay) NA 



c. Stufflebeam's initial 

presentation NA 1 

(lo: 

d. Reinhard's initial presen- 
tation (advocacy teams) NA 1 



Scriven *s "special topics" 
presentation (goal -free 
evaluation) NA 



f, Stufflebeam's "special 

topics" presentation (School 

System evaluation) NA 1 



g. Resource table (materials) NA 1 



h. Informal discussion with 

peers NA 1 

o; 

i. Discussion of participants' 
evaluation problem (Fri. a.m.) NA 1 



General Question & answer 
sessions with staff NA 



Discussion of workbook 
examples NA 
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1. Thursday morning reading 

session (9-12) NA 



Note: Frequencies are shown in parenthese. 



2 
(2) 


3 
(9) 


4 
(4) 


5 
(1) 


3.00 


1.03 


2 

(7) 


3 

(7) 


4 


5 

(1) 


1.91 


.85 


2 

(11) 


3 

(2) 


4 


5 


1.65 


.65 


9 
L 

(4) 


2 

(8) 


A 
H 

(7) 


C 

0 

(2) 


3.33 


.91 


2 

(12) 


3 

(2) 


4 


5 


1.67 


.64 


2 

(8) 


3 


4 


5 


1 CO 

1 . 5o 


. /c 


2 

(5) 


3 

(5) 


4 

(6) 


5 

(1) 


2.71 


1.33 


2 

(11) 


3 

(5) 


4 

(4) 


5 


2.54 


1.06 


2 

(10) 


3 

(8) 


4 

(1) 


5 


2.23 


.83 


2 

(11) 


3 

(7) 


4 

(2) 


5 


2.29 


.86 


2 

(5) 


3 

(4) 


4 

(8) 


5 

(1) 


2.96 


1.23 


2 

(2) 


3 

(6) 


4 
(6) 


5 

(5) 


3.30 


1.33 
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during the Institute and in the open-ended questions, that time spent in 
reading was wasted. It was suggested that advance reading materials be 
sent so that all required reading could be done prior to the Institute, and 
that the morning session be replaced by more time in staff -directed 
activities. This, of course, assumes that participants will in fact read 
any materials sent to them in advance. However, if they fail to do so, 
the failure is at least of their own doing. 

The resource table £er se was rated slightly more positively, and a 
number of participants were observed browsing through the materials during 
free time throughout the two days. Several others requested that the 
resource table be available at all times not spent in specific Institute 
activities (e.g., at the conclusion of the evening session). 

Activities rated as most valuable included Scriven*s presentations on 
the "Pathway" model of evaluation and goal -free evaluation, and Stufflebeam's 
presentations of problems faced by evaluators and school system evaluation. 
A few people expressed the desire to have more time set aside to interact 
with each of these ideas before moving on to another topic. 

Activities rated least valuable were Scriven's talk on school 
accreditation, Reinhard's discussion of advocacy teams, and the Thursday 
evening workbook exercises. The accreditation lecture came late in the 
Institute (Friday afternoon); many participants seemed generally tired or 
disinterested, which may or may not have been a function of Scriven's 
presentation. Similarly, Reinhard*s presentation on advocacy teams came 
toward the end of a two-hour session and might have benefitted if it had 
been scheduled after a break or at some other more favorable time. Also, 
this was the only one of the three initial presentations in which questions 
were consistently asked during the lecture instead of at the end; this seemed 
to detract from its continuity. 
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Although ratings of the workbook examples ranged from very valuable 
to worthless, the workbook produced numerous negative com-nents from pnrtici- 
pants during Thursday dinner. The staff's directions were perceived by both 
the evaluators and by several participants to be unclear; participants 
appeared to be both confused and hostile toward the activity as a whole. 
However, the directors managed to salvage the activity reasonably well 
during the course of the evening session. 

Discussion of participants* evaluation problems and general question 
and answer sessions with staff were rated as valuable. A majority of the 
participants rated the informal discussion with peers as valuable, even 
though little time was available in the schedule for interaction among 
participants. * 

Responses to the open-ended questions seemed to be slightly less 
positive than those to more structured items. Responses to open-ended 
questions which tended to be recurrent among participants are summarized 
in Table 12. Several participants indicated both verbally and in their 
responses to the open-ended questions that they would have preferred 
more lecture and discussion emphasis on internal rather than external 
evaluation. Others felt they would have likad greater diversity in the 
"alternative conceptions of evaluation" presented. Some persons said they 
had expected Scriven and Stufflebeam to hold quite different views on 
evaluation, but found that they were not sufficiently different in their 
approach to evaluation to generate useful and provocative dialogues. This 
criticism seems relevant, since the Institute was billed as alternative 
conceptions of evaluation, as is noted in the "flyer" shown herein as 
Appendix H. 



Table 12 



Summary of Participant Perceptions of Major Strengths and Weaknesses 
of the Evaluation Institute, from Responses to Open-ended Questions 



Strengt hs 

1. Interactions between staff and 
participants. 

2. Qualifications and reputations 
of staff. 

3. Presentation of new perspectives 
on various issues in evaluation. 



Weaknesses 

1. Lack of advance organization 
(e.g., statement of prerequisites, 
advance reading materials). 

2. Thursday morning reading session. 

3. Not enough time for personal 
interactions with staff. 

4. Excessive emphasis on external 
rather than internal evaluation. 

5. Daily sessions too long. 

6. Lack of diversity in models 
and concepts discussed. 



General evaluation of the Institute * Overall participants' reactions 
to the Institute are summarized in Table 13. The majority of the partici- 
pants believed that what they had learned from the Institute would be useful 
to them in their work. A majority of participants also stated both that 
they would attend this Institute if they had to choose again, and would 
recommend it to a colleague. 

Reactions to various aspects of the Institute are presented in Tables 
14, 15, and 16, with the employing institution of participants used as a 
stratifying variable. These analyses were efforts to determine if partici- 
pants from universities differed in their reactions from participants from 
public schools, etc. However, there appear to be no discernable differences 
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among groups on this stratifying vr.riahle with respect to responses con- 
cerning the topic, tuition fee, and usefulness of the Institute. 

Tabulations of responses to the staff questionnaire are available in 
Appendix I. These seem to speak for themselves and will not be discussed 
here. 

Table 13 

Tabulation of Participant Responses to Items on the 
General Evaluation of the Institute 

Item 

23. Do you anticipate that what you have learned from 

the Institute will be useful to you in your own work? 

definitely 1 2 3 4 5 definitely not 
(11) (6) (5) (2) 

27. If you were able to choose over again, would you attend 
this Institute? 

(17) Y es (2) N o (5) Uncertain 

28- If this Institute were held again, would you recommend 
to a colleague that they attend? 

(17) Y es (4) N o (3) Uncertain 



X s 



1.92 1.02 



Note: Frequencies are shown in parentheses. 
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Table 14 

Summary of Responses on Item 4, stratified by 
Employing Institution 



Item 4: Leaving aside the quality of instruction for the moment, do you 
think the topic treated in this Institute should be treated 
again in future Institutes? 



Employing 
Institution 


definitely 
1 2 


3 


definitely 

not 
4 5 


Total 


College or 
University 


(58) 


4 

(34) 


1 

(8) 


0 

(0) 


0 

(0) 


12 
(100) 


r UD 1 1 C oCnOO 1 

System 


6 

(86) 


1 

( 14) 


U 

(0) 


0 

(0) 


0 

(0) 


(100) 


Regional Labora- 
tory 


1 

(100) 


0 
(0) 


0 
(0) 


0 

(0) 


0 

(0) 


1 

(100) 


Federal Government 


1 

(100) 


0 

(0) 


0 

(0) 


0 

(0) 


0 

(0) 


1 

(100) 


State Dept. of 
Education 


1 

(100) 


0 

(0) 


0 

(0) 


0 

(0) 


0 

(0) 


1 

(100) 


Other 


1 

(100) 


0 

(0) 


0 

(0) 


0 

(0) 


0 

(0) 


1 

(100) 


Total 


17 
(74) 


5 

(22) 


1 

(4) 


0 

(0) 


0 

(0) 

- ■ 


23 
(100) 



Note : Percentages of row totals are shown in parentheses below each 
frequency. 
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Table 15 

Summary of Responses on Item 23, Stratified by 
Employing Institution 



Item 23: Do you anticipate that what you have learned from the Institute 
will be useful to you in your own work? 



Employing 
Institution 


definitely 
1 ^2 


3 


definitely 

not 
4 5 


Total 


College or 
University 


5 

(42) 


(25) 


3 

(25) 


(8) 


U 

(0) 


12 
(100) 


Public School 
System 


3 

(43) 


2 

(29) 


2 

(28) 


0 


0 


7 

(100) 


Regional Labora- 
tory 


0 

(0) 


0 

(0) 


0 

(0) 


1 

(100) 


0 

(0) 


1 

(100) 


Federal Govern- 
ment 


1 

(100) 


0 

(0) 


0 

(0) 


0 

(0) 


0 

(0) 


1 

(100) 


State Dept. of 
Education 


1 

(lOG) 


0 

(0) 


0 

(0) 


0 

(0) 


0 

(0) 


1 

(100) 


Other 


1 

(100) 


0 

(0) 


0 

(0) 


0 

(0) 


0 

(0) 


1 

(100) 


Total 


11 

(48) 


5 

(22) 


5 

(22) 


2 

(8) 


0 

(0) 


23 
(100) 



Note: Percentages of row totals are shown in parentheses below each 
frequency. 
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Table 16 



Summary of Responses on Item 24, Stratified by 
Employing Institution 



Item 24: Considering what you have learned by attending this Institute 
(or any other benefits you have received from attendance), 
the tuition fee was: 



Employing 
institution 


considerably 
too low 

1 2 


3 


considerably 
too high 
4 5 


Total 


College or 
Uni varsity 


0 


0 


8 

(57) 


1 

(8) 


3 

/ OCX 

(25) 


12 
(100) 


Public School 
System 


0 

(0) 


0 

(0) 


4 

(57) 


3 

(43) 


0 

(0) 


7 

(100) 


Regional Labora- 
tory 


0 

(0) 


0 

(0) 


1 

(100) 


0 

(0) 


0 

(0) 


1 

(100) 


Federal Govern- 
ment 


0 

(0) 


0 

(0) 


0 

(0) 


1 

(100) 


0 

(0) 


1 

(100) 


State Dept. of 
Education 


0 

(0) 


0 

(0) 


(100) 


0 

(0) 


0 

(0) 


1 

(100) 


Other 


0 

(0) 


0 

(0) 


1 

(100) 


0 

(0) 


0 

(0) 


1 

(100) 


Total 


0 

(0) 


0 

(0) 


15 
(65) 


5 

(22) 


3 

(13) 


23 
(100) 



Note: Percentages of row totals are shown in parentheses below each 
frequency. 
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Summary and Recommendations 

Overall, participants' reactions to the first "Alternative Conceptions 
of Evaluation" Institute were quite positive. The directors and staff also 
felt the Institute was generally successful. These reactions are congruent 
with those of the evaluators. It is the judgment of the present authors 
that the Institute was, in general, a success both as an individual two- 
day session and also as the first in a proposed series of three AERA 
traveling institutes on evaluation. The evaluators strongly recommend 
that this series of institutes be continued. However, there are a few 
suggestions which might be made for the improvement of the January 
Institute. 

1. The Institute objectives and detailed agenda should be made 
available to participants sufficiently in advance of the Institute that 
they know what to expect. It should either be made more clear that the 
two days are not to be spent in personal consultation with each individual, 
or more time should be scheduled for this sort of activity. 

2. Information about participants' prior training and experience in 
evaluation should be collected and reviewed by the Institute directors to 
assure an appropriate match between participants* backgrounds and the con- 
tent and activities of the Institute. In the present Institute, the staff 
tended slightly toward the perception that the content was too difficult 
for participants, while a large majority of the participants felt they 
were over-prepared for what they experienced. Resolving this particular 
discrepancy in perceptions is not important, but it does point to a need 
to attend to the match between background and activities. (The inability 
to resolve the discrepancy also points to the need for a better measure 
of learning for subsequent institutes.) 



3, The next Institute should either present more closely delineated 
alternativ e conceptions of evaluation or the billing of the Institute 
should be changed to avoid misleading prospective participants, 

4, Based on the reactions of participants, it might be recommended 
that the Institute should be lengthened to three days. However, there is 
no way to determine from these data whether participants would be willing 
to pay a proportionate increase in fees. In view of current ambiguities, 
surrounding fee policies for AERA training sessions, this recommendation 
might best be viewed as something which would be desirable but may or may 
not be feasible. Until more data are collected on more AERA training 
sessions under the present fee structure, the Institute directors and 
AERA Model A director will simply have to make a best guess on this. 

In the evaluators' opinion^ more time would prove very useful in helping 
participants satisfy their objectives in attending the Institute, 

5, The first morning reading session should be abandoned. Instead, 
all necessary materials (e.g, , notebooks) should be mailed to participants 
sufficiently prior to the Institute that they have time to read them in 
advance. The reading session should either be (a) replaced with extra 
structured activities, (b) split to provide time to use the "resource 
table" throughout the two days, or (c) activities should be advanced by 

a few hours and the evening work session should be abandoned, 

6, More time could be given to informal interaction among partici- 
pants, either with or without staff present, to enable them to become 
acquainted and to discuss issues presented. However, this should be done 
only if it can be scheduled without sacrificing any time now spent in 
directed activities. For example, if the morning reading session were 
cancelled, activities could be moved ahead a few hours, leaving the 
evening free for more informal interaction. 
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7. Instructions should be presented more clearly. This is especially 
true for the workbook activity, if it is to be used again. 

8. More work should be expected of participants during the Institute. 
The data which support this recommendation may merely reflect the prevalent 
belief structure that passive reception is a less effective mode of 
learning than more active alternatives. Whatever the source, participants 
expressed a feeling that they were not worked hard enough a rare complaint 
in the experience of the evaluators. Ways should be found to involve 
participants in relevant reading, simulations, or activities which demand 
participant work on evaluation designs, instruments, etc., to supplement 
verbal input they receive. 

9. Presentations and activities should be more carefully integrated to 
make the overall presentation more cohesive. 

10. Internal evaluation should be emphasized more and external 
evaluation of programs emphasized less in subsequent institutes (unless 
different types of participants are selected). Informal comments made to 
the evaluators support participants* written reactions and suggest strongly 
that they are not in attendance to learn how to evaluate someone else's 
program, but have to evaluate their own. Of course, there were some 
exceptions to this statement and a different group of participants could 
differ dramatically from the present group on this dimension. Perhaps 
a tailored question on the application form could serve a useful purpose 
in sensitizing the directors to variation on this dimension or may even 
enable them to select participants who have common needs relating to the 
Institute content. 



11. A final recommendation which is supported only very indirectly 
by the data reported herein is that the number of participants in the 
next Institute should be increased.^ In the opinion of the evaluators, 
there is nothing in the present Institute format or in the recommendations 
for modification listed above which would in any way prevent 30 or 35 
participants from receiving training equal to that received by the 
present participants. Given the fact that one purpose in sponsoring 
the series of traveling institutes is to determine whether or not AERA 
could conduct such activities on a self-supporting (or even profit- 
making) basis in the future, it would be highly desirable to vary the 
number of participants admitted to the institutes. The Institute 
evaluated herein cost AERA a total of $2,562 (including direct costs for 
the Institute and evaluation and indirect AERA Central Office costs for 
the former) and yielded a return of $2180 in participant fees, for an 
overall loss of $382. Although this is not serious under the present 
situation where such experimentation Is underwritten by USOE, It Is 
Important to determine now whether Institutes could be run in the future on 
a break-even or profit basis without sacrificing quality of training. 
To the extent that this recommendation could be implemented, it would 
be helpful in the forthcoming overall evaluation of the practicability 
of AERA sponsoring traveling institutes without Federal support. 



This recommendation is at mild variance with staff perceptions 
relating to number of participants (see Appendix I, Item 3). 



Appendix A 
Tentative Agenda 
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AERA TRAINiriG lt;STlTUTE 
TENTATIVE AGENDA 



October 19> 1972 
9:00 - 11:00 



Registration and independent reading 



11:00 - 11:30 
11:30 - 1:00 
1:00 - 5:00 



Evaluation 



Lunch 



Perspectives on the field of evaluation 
and discussion of particular evaluation 
problems 



Blaine Worthen 



Michael Scri ven 
Dan iel Stu^f lebean 
Diane Reinhard 



Di scussion 



Introduction of Jin Sanders' materials 
on evaluation oroblems 



Organization of small groups 



5:00 - 7:00 
7:00 - 10:00 



Dinner 



Case work by groups focusing on selected 
models 



Discussion 



Cases without answers 



Discussion 



10:00 



Adjournment 



Oc tc>h::r 70. V -qi 



9:00 - 11:30 Discui;sion of cofr.non elements Scrivcn 

in evaluation *. Stufflebeam 



Group work focused on problems posed 
by participants. (Within each group 
one member would serve as a client, 
one* as an evaluation design specialist, 
one as a recorder, and one as a general 
evaluation administrator. The evaluation 
group would attempt to respond to the 
evaluation problem posed by the person 
playing the role of the client). 

11:30 - 1:00 Lunch 

1:00-3:00 Group reports 

Discussion of the reports 

Discussion of the institute 
3:00 - 3:^5 Final evaluation Blaine Worthen 



3:^5 



Adjournment 
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Application Form 



APPLICATION FOK AERA RESEARCH TRAINING INSTITUTE 
ALTERNATIVE CONCBPTIOMS OF EVALUATION 
Michael Scriven & Daniel Stufflebeam 



GEUIRAL INFORMATION 

1. Ncune: ^ 

Last First Initial 

2. Mailing address: 



3. Present Institutional Affiliation (e.g,|UCLA, NYC. School DistJ; 



4. TITLE: 



EMPIOYMENT INFORMATION 

5. Describe briefly the nature of your present employment: 



6. Describe briefly any changes you expect in your employment during the coming 
year with respect to either employer or type of activity: 



7- What percent of your time is allotted to teaching? b. To Research or 

Evaluation? c. To grad. study? d. Administration? 



EDUCATIONAL HISTORY 

8. Masters School: b. Doctoral School: 

Major Major 

9. Record in the blank the approximate number of courses you have taken at either 
undergraduate or graduate level in each of the following areas: 

Curriculum Psychology (Exper . , Soc . , Devel . # or Learning) 

^ Educ. Administration Sociology 

Educ. Measurement or Psychometrics Computers 

Statistics and experimental design 

PROrESSIONAL AND SCHOLARLY INTERESTS 
10. Approximately how many research articles which you have authored alone or jointly 

have been accepted in scholarly (refereed) journal? ^_ 

List no more than three professional societies other than AERA of which you 
are a member 



In order to insure that this Institute is responsive to the interests and needs 
of the participants please indicate, on a seperate sheet(s) , the relevance of 
this Institute to your present responsibilities, your interests, problems, and 
hopes for and in the field of evaluation, and the specific personal objectives 
you hope to realize by attending this training institute. 

Return this application and two copies with a self addressed stamped envelope 
to American Educational Research Association, Training Institute, 1126 Sixteenth 
Street, N.W., Washington, D.C. 20036 



liot-: Do not send registration fee with this 
O „ application 
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All documents listed on the previous pages have been nunbered and placed 
on the table within certain cluster areas*. If you are looking for a specific 
document, just scan the following matrix to see v/here the document is located, 
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65. **A Proposal for an Academic Faculty of Evaluation," The Ohio State 

University Evaluation Center, College of Education, Columbus, 
Ohio, September 10, 1972. 
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Examples of Evaluation Instruments, Questions Addressed by Those 
Instruments > and Information Needed to Answer Those Questions. 



Participant Questionnaires 

A. On-site 

1. What type of audience is attracted to the Institutes? 

a. Professional roles 

b. Experience related to the topic 

c. Distance of home from Institute site 

d. Professional credentials 

e. Payment of expenses 

f. Attendance at AERA convention 

g. AERA membership 

2. Was the Institute well organized? 

a. Publicity and pre-information 

b. Qualifications of staff 

c. Advance learning materials 

d. Appropriateness of fees 

e. Convenience of site and dates 

f. Importance of the chosen topic to the profession 

3. Was the presentation of the topic interesting and effective? 

a. Content validity 

b. Provision for individual differences 

c. Scheduling 

d. Variety and balance of presentation 

e. Opportunities for interaction with staff 

f. Length of Institute 

g. Supplementary materials 

h. Statement of objectives 

4. Did participants achieve positive gains from attending the Institute? 

a. Cognitive gains 

b. Affective gains 

c. Achievement of personal objectives 

d. Acquisition of professional contacts 

5. Were the facilities and equipment at the Institute satisfactory? 

a. Meeting rooms 

b. Instructional equipment 

c. Supplementary resources 

6. Were the evaluative procedures employed effective? 

a. Formative 

b. Participant feedback 

c. Resulting changes 

d. Resulting disruption 

e. Comparison with pre- and post-sessions 

B. Follow-up 

!• Did the participants achieve positive gains from attending the 
Institute? 
a. Affective gain 

b« Professional utilization of acquired skills and knowledge 
c. Perception of competency growth 
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II. Staff (On-Site) 

A. Was the orcjanization of the Institute satisfactory? 

1. Selectic)]) of audience 

2. Cooporaticn frcni Central Office 

B. In wiiat Wc\ys did staff p*:irceive participants as benefitting from 
attending the Institute? 

C. Were the facilities and equip i^ent at the Institute satisfactory? 

1 . Meeti ncj rooms 

2. Livincj arrangements 

D. Were the evaluative procedures employed effective? 

1. Helpfulness of formative evaluation 

2. Cooperation from evaluate rs 

3. Resulting changes 

E. Comparison with pre- and post-sessions. 

III. Questionnaire for applicants who failed to attend 

A. What type of audience v/as initially interested in the Institute but 
failed to attend? 

1. Professional role 

2. Distance of home from Institute 

3. Professional credent i als 

4. Attendance at AERA conventions 

5. Experience related to the topic 

B. Why did they choose not to come? 

1. Qualifications of chosen staff 

2. Fee structure 

3. Other expenses which would be incurred 

4. Convenience of site and date 

5. Professional relevance of the chosen topic 

6. Other reasons (open-ended) 
IV, Non-applicants 

A. What type of AERA member expressed no interest in (or was unaware of) 
the Institute? 

1 . Professional roles 

2. Experience related to the topic 

3. Distance of home from Institute site 

4. Professional credentials 

5. Attendance at AERA convention 

B. Were they aware of the Institute? 

1 . Publi ci ty 

2 . Pre-information 

C. If so, why did they not apply? 

1. Qualifications of chosen staff 

2. Fee structure 

3. Convenience of site and dates 

4. Relevance of the chosen topic 
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Appendix F 
Instruments 



1972 A'-Pii TrAVolin^; In:.titui.. 
Participant la-stirument: //I 

1. Topic of the Institute you are attending: 



2, Location of the Institute yon ate attending (state) 



You are presently employed in a: (check the ONE that is most applicable) 

College or University 

Public School System 

Educational K&D Center 

Regional Laboratory 

Federal Governinant 

State Department of Education 

I am prar^>ently a student 

Other (please specify) \ 



Your primary role at your employing institution is that of; (check the 
ONE that is most applicable) 

teacher 

adminiGtrator 

researcher 

evaluator 

developer 

supervisor 
student 

^..^^^^ other (please specify) 



5. What academic degrees do you hold? (check the most advanced degree held) 

bachelor's 

^.^^^^^^ master's 

doctorate 

other (please specify) 



6* Please check the Divisionvs) of AERA of which you are presently a member. 

A (Administration) 

, B (Curriculum and Objectives) 
. C (Learning and Instruction) 



L) (Meaouremont and Research Methodology) 
_ E (Counseling and Human Development) 
^ F (History and Historiography) 
_ G (Social Context of Education) 
^ H (School Evaluation and Program Development) 
^ Student Motnber 

_ I am not presently a member of AERA 
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7. How did you Hr^^ l<?.aru aboui: uiLi Lty.iticulii't (check Oi^:) 

profoi>sl:3U.'il f ublicatioa (i/leafic spacify)_^_ 

colloc'i^ue r>r rtudont 
nnllcd not: 

other (plea?;ci specify) 

8. What are your primary reasons :'or attotiding this section? (check nil 
tht'it apply) 

. desire to gain new skills or knowledge related to the topic 

desire to inprove exi6ti:^r. skills or knowledge related to th2 topic 

desire to discuss probleiiis related to the topic with expercs 

other (please specify) 



9. What prior experience have you had with the topic of this Institute? 
(check one) 

no experience whatsoever 

have had limited experience (e^g,, have read a little about the topic) 

have had considerable experience (e.g., took a clasa on the tojjic, 

or worked for several months on activities where the topic was used) 
great amount of experience (e.g., have used the topic for several 

years, or have taken several classes on the topic) 

10. If this session were not cwailable now, but was offered inste?id as a seDZ?oi\ 
for a similar amount of time either immediately before or after the cokvcmh i on 
in New Orleans in February, would you choose to attend the session there? 
(Assume registration fees, etc*, remained constant) 

Yes No 



11. As far as you now know, do you plan to attend the AERA Conv ention in Nj-v; 
Orleans this year? 

Yes No 



12. Is the date selected for the Institute convenient for you? 
Yes No 



13. Is the site selected for the Institute convenient for you? 

^Yes No 



lA , How far is your home from the site of this Institute? (check ORE] 

less than 50 miles 

50-100 miles 

101-500 miles 

] 501-1000 miles 

more than 1,000 miles 
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]5. Appro .•.l..n.tP..ly what porcenta,,a of your cost., in each ca.e;-ory below did 
o^oloJL) '^"^^'^^^ "ill l^'^ rc:i.bun-o.d by your 



travel 
_Z room and board 
__% tuition fee 
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AERA Traveling Institutes 
Participant liistrument //2 



(ID tsuiiiber) 



Institute: 



Location: 



Please circle the code number v/hlch beat describes your level of understandinr, 
of each of the topics below: 



Code 

1 
2 

3 

/. 

~t 

5 



Interpretation 

Excellent understanding 
Good understanding 
Some understanding 
Little understanding 
No understanding 



Topic 



Excellent Good Some Little None 



3. 
4. 
5. 



8. 
9. 

10. 
11. 



Problems evaluators have to 
deal with 

AlteLaative frameworks for 
dealing with evaluation problems 
(e.g., Stake's Countenance Model, 
Provus' Discrepancy Model) 

Criteria for meta-evaluation 

Definition of evaluation 

Cost considerations in evaluation 

Evaluation designs, including time 
series vs . control groups 

Procedures for site visit 
evaluation 

The role of values in evaluation 

Emergent techniques for working on 
evaluation problems (e.g., advocacy 
team approach) 

Evaluation methodology 

How to implement a strategy for 
solving evaluation problems 



1 
1 
1 

1 



12. Roles of the evaluator 



1 

1 

1 
1 

1 
1 



2 
2 
2 
2 



2 
2 

2 
2 

2 
2 



3 
3 
3 
3 



3 
3 

3 
3 

3 
3 



4 

4 
4 



4 
4 

4 
4 

4 

4 



5 
5 
5 
5 



5 
5 



AERA Traveling Institutes 
Participant Instruiriciiit il3 

.PJli'5Aif!?ii- "^^^^^ cpJnionnalre attempts to £sr*ess attiti^dec of participants 
tov;ard educational evaluation, Pleace indicate your agreement or disagreement 
with cncli statement by checking (/) the answer that best describes how you 
personally feel, re:>nrdless of whether oi-her people may agree or disTigree 
with you. Although many nf the. questions may appear similar, please judge 
each one on an individual basis. Since we necid to know your attitude, please 
ar.swer each queation frankly and honently. There are no correct responses. 
There is no time limit, but respond as quickly as you can, and do not: leave out 
any of the statements . 

Topic of Institute ^ 

Location of Institute 



!• I see and feel deeply a need for increased evaluation information for 

decision-making in my field, 

( ) ( ) ( ) ( ) ( ) 

Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree 



2. Educational evaluation usually results in arbitrary judgments about the 
educative process. 

( ) ( ) ( ) ( ) ( ) 

Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree 



3. There is cuiiently too much concern with evaluation in education, 

( ) ( ) ( ) ( ) ( ) 

Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree 

A. Educators do not receive sufficient training in evaluation, 

( ) ( ) ( ) ( ) ( ) 

Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree 



5, Educative processes are too complex to be evaluated objectively. 

( ) ( ) ( ) ( ) ( ) 

Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree 



-2- 

0, jivnlualiun Is an attCMauL to rcfcluco education co a r«e.c}icuiis tic proc.e5ia. 

( ) ( ) ( ) < ) ( ) 

Strongly Disa^jree IhulecidoU or Agrpe Strongly 

Diaa^cce Neutral Agree 

?• Educational evaluation usually rtisultfj in improvements in educational 
practice. 

( ) ( ) ( ) ( ) ( ) 

Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree 

8. Evaluation should aid an educator in revising his goals even while 
the program is in progress. 

( ) ( ) ( ) ( ) ( ) 

Scrongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree 

9. Evaluation interferes with the running of schools more than It helps. 

( ) ( ) ( ) ( ) ( ) 

Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree 

10. Intuition and general knowledge of practitioners are more valuable than 
formal evaluative procedures in making decisions in education. 

( ) ( ) ( ) ( ) ( ) 

Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree 

11. Using educational evaluation in my work does trOt appeal to me. 

( ) ( ) ( ) ( ) ( ) 

Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree 

12. Money spent on evaluation contributes more to the improvement of 
education than any other expenditure. 

( ) ( ) ( ) ( ) ( ) 

Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree 



ERIC 



AERA Traveling Institutes numner 

Participant Inntruinent //4 

^^^^ evaluation form is administered by the AilVA Research Training 
COuv;.it;cc'o or per8onsfi3biytiug them with thie evaluation. Completed formn will 
be returned directly to the Coinmittec by the person administering this form. 
After the data have been tabulated, the Instructior.al staff of your session 
may request a sunanary. The principal purpose of this form is to assist in the 
pla ning of subsequent traveling institutes. Therefore, be completely candid 
in your responiies. Do not sign your name. 

Topic of this Institute 

Location of this Institute 

For each question or partial statement below, please circle the number which best 
reflects your reaction. 

1. How important do you feel the topic of this Institute is to educational 
research and/or evaluation? 

very important 1 2 3 4 5 very unimportant 

2. How important do you feel the topic of this Institute is to educational 
practice? 

very important 1 2 3 4 5 very unimportant 

3. How relevant was the content of the Institute to the topic that was 
advertised? 

highly relevant 1 2 3 4 5 not at all relevant 

4. Leaving aside the quality of instruction for the moment, do you think 
the topic treated in this Institute should be treated again in future 
Institutes? 

definitely 12 3 4 5 definitely not 

5. The staff's objectives for this Institute were: 

very clear to me 1 2 3 4 5 very unclear to me 

6. Tlie planning and organization of this Institute was: 
excell ent 12 3 4 5 poor 

?. Overall, was the Institute long enough to cover the topic adequately? 
considerably too long 12 3 4 5 considerably too short 

8. As a rule, daily sessions were: 

considerably too long 12 3 4 5 considerably too short 
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9. Do you feel all laecesaary prerequisites were clearly stated in infonaation 
you received prior to the Institute? (check ONE) 

Yes, they wore clearly stated 



_No, they were ambiguous 

Jlo prerequisites were lloted 



10. Did you receive advance reading materials from the Institute director (s) 
early enough to read them prior to the Institute? (check ONE) 

Yea, materials came early enough 



_^Nc, materials caiae too late 
No materials were sent 



11. Do you think you entered the Institute with the appropriate prerequisites or 
prior knowledge to make what you learned there of value to you? 

I had more than ^ ^ 2 ^ 5 ^ seriously 

enough preparation lacking in preparation 

12. How often did the instructional procedures take into account variability 
in prior knowledge brought to the Institute by participants? 

always 1 2 3 4 5 never 

13. Overall, the quality of instruction in this Institute was: 
excellent 1 2 3 4 5 poor 

14. The instruction was generally: 

too lecture-oriented 12 3 4 5 too discussion-oriented 
very interesting 12 3 4 5 very uninteresting 

very informative 1 2 3 4 5 very uninformative 

15. Opportunities for asking questions and studying were: 
sufficient 12 3 4 5 insufficient 

16. The amount of work the staff required of you was: 

far too much 1 2 3 4 5 far too little 

17. Opportunities for you to interact with the staff with respect to problems 
in your own work which relate to the topic were: 

sufficient 12 3 4 5 insufficient 

18. In your opinion, the staff members were in general: 

very well-qualified 12 3 4 5 very unqualified 
very well-prepared 12 3 4 5 very unprepared 
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19 • Did the staff f;eek your reactions to their instructional procedures, 
Gchedu.ling, etc., (luring the Institute? 

frequently 1 2 3 ^4 5 never 

20, Did It appear to you that your reactions led to improvements in the 
ins tructiona.1 procedures, schoululing, etc.? 

frequently 12 3 4 5 never 

21. The formal evaluation of this Institute by the "outside evaluation" 

(e.g., those instruments) was: 

not at all disruptive 12 3 4 5 very disruptive 

22.. The meeting room facilities for the Institute were: 
excellent 12 3 4 5 poor 



23. Do you anticipate that what you have learned from the Institute will 
be useful to you in your own work? 

definitely 1 2 3 4 5 definitely not 



24. Considering what you have learned by attending this Institute (or any 
other benefits you have received from attendance), the tuition fee was: 

considerably too low 1 2 3 4 5 considerably too high 

25. Approximately how much money (including travel costs, living expenses, 
and tuition) did this Institute cost you personally (i.e., was not 
reimbursed) ? 



26. What was the total amount of money (including travel, living expenses, 
and tuition) your attendance cost, including personal coats to you and 
coats paid by your institution or other sources? $^_^^ ^ 

27. If you were able to choose over again, would you attend this Institute? 

Yes No Uncertain 



28. If this Institute were held again, would you recommend to a colleague 
that they attend? 

Yes No Uncertain 
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29. How would you rntie tlio value of each of the following aspects of 
this Institute? 

(Please circle ONE for EACH row) 

no opportunity ^ery 

to judge valuable worthless 

a, Scriven on accreditation NA 1 2 3 4 5 

b, Scriven*s initial presentation NA 1 2 3 4 5 
(Pathway) 

c, Stuf f lebeams 's initial presentation NA 1 2 3 4 5 

Reinhard's initial presentation NA 1 2 3 4 5 

(Advocacy teams) 

e, Scriven' s "special topics" presen- NA 1 2 3 4 5 

tation (goal-free evaluation) 

f« Stuf f lebeam's "special topics'* pre- NA 1 2 3 4 5 

sentation (School System evaluation) 

g. Resource table (materials) NA 1 2 3 4 5 

h. Informal discussion with peers NA 1 2 3 4 5 

i. Discussion of participants' NA 1 2 3 4 5 
evaluation problem (Fri, a.m.) 

j. General question and answer sessions NA 1 2 3 4 5 

with staff 

Discussion of workbook examples NA 1 2 3 4 5 

1, Thursday morning reading session NA 1 2 3 4 5 

(9-12) 



30, Please list any aspects of the Institute other than those listed in 29 
above which you felt were of considerable value. 



31, Please list any criticisms or suggestions you have concerning time 
allocation for any of the Institute activities or sessions. 



32, a. Please list any elements which vere missing from this Institute that 
you feel would be of value in future Institutes of this type* 



b, Wliat would you suggest sacrificing from the present Institute format 
to make room for new topics or activities? 



ERIC 



5 



33. Based on your e::pcripnce v/lth this Institute, what do you see r.s the 
Uiajor advant:af^,cs or diGc:dvantcu'.r'js of the Trdvollng Institute concept, 
cop^r^^rGd v/ith the usjual AElvi aiinual mcetins pre- a*7d postset'sion? 

Major ad v ant'a c ;es of Traveling Tnatltutea : 



Major disadvantages of Traveli n g Institutes : 



34. Please write any additional coircnents and/or suggestions below or on the 
reverse side. 



197?. AERA Travelln,^ IntiLitutos 
Staff Instrument ill 



Topic of Institute ^I.ocation^^ 

As pnrt of our evaluation of thei AjLM Traveling Institutes, ve feel it 
important to obtain your rcactiors, as directors or si^.ff tDen.bers, to several 
aspects of the Institute. V7e will appreciate your candor in rcf^ponding. 

For each partial statement printed bc-J.ow, plea&e circle the VvUM-iber which 
best reflects your reaction to that statement. 

1. The meeting rooms were; 

excellent 1 3 4 5 poor 

2. The living accomodations were: 
excellent 12 3 A 5 poor 

3. The number of participants was; 

too large 1 2 3 A 5 too small 

4. The background of the participants (as a group) for the topic of this 
Institute was: 

very adequate 12 3 4 5 very inadequate 

5. As a group, motivation and interest of the participants appeared to be: 
very high 1 2 3 4 5 very low 

6. In terms of following your (or the director's) instructions and schedule, 
participants were: 

very cooperative 12 3 4 5 very uncooperative 

7. The next Traveling Institute on this topic should be: 
considerably longer 12 3 4 5 considerably shorter 

8. For this group of participants, the content presented was: 
too easy 12 3 4 5 too difficult 

too theoretical 12 3 4 5 too non-theoretical 
very useful 12 3 4 5 useless 

9. Participants' knowledge and/or skills related to this topic have: 
Increased considerably 12 3 4 5 shown no increase 



10. As a facilitator in making arrangements for chis Xastitutc^ lha efforts 
of AEKA Central Office siiaff vere: 

very helpful 1 2 3 4 !i not at all helpful 

11. Formative evaluation feedback proved: 

very helpful 12 3 4 3 not at all helpful 

12. As a result of formative evaluation 'feedback you made: 
many changes 1 2 3 4 5 no changes 

13. Overall, the evaluation procedures were: 

not at all disruptive 12 3 4 5 very disruptive 

14. Overall, your objectives for th:.s Institute were: 
attained 1 2 3 4 5 not attained 

(Please be as specific as possible in responding to the open-ended questions below.) 

15. Now that the Institute is over, what things do you feel went especially veil? 



1.6. What areas do you feel are most in need of improvement before next time? 
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17. LaiJt:d on your experic-u-iCi.' vd.th the /J:KA 'i'j:fiveHrji': lnr?t.i L»jt. <' --o faCj 

whal: do you sec af: uliaiir rntijor .idvant,- and disndv.MiL.-jf^r-*' as ^' iir'i^«ir^-A 
wit h t'rie usual AIIR.^. pre— and ^.r r.L-- ccc.fi.^o'Ui? 

Aivantaea^ of T r avrvl ing j as t i i^. : 5;^ : 



Dlsadvantafias of Traveling Institutes ; 



18. Any other coments on ways to Improve the Institutes or the procedures 
for evaluating the Institutes? 



Appendix I 
Tabulations of staff Questionnaire 



1972 AuRA Tv^vvui 



Staff Inbf.ruiiont ?/l 



Toi»Ic of ln£>titute J.ocaLi.oii 

part, of our cv/aluacloii of tli.; AtllA 'Cr-'i/r: ].-].:);> Tn;juiiuLes, v;c feo*! it 
djnporcanU to obtain your re:ictions, as director?: or siciff tjerubcrs, to several 
aspciCta of the Institute. W»;i vlll appreciate >our candor la riispondin^:. 

For each partial y tato.iucai: printed br^l.ow, piease circle the nuniber v/hich 
best reflects your reaction to i.hcit statein^int. 

1. The meeting rooms were: 

excellent 1 2 3 4 5 poor 
(2) (1) 

2. The living accomodations were: 

excellent 1 2 3 4 5 poor 
(3) 

3. The number of participants was; 

too large 1 2 3 4 5 too f^rnall 

(1) (2) 

4* The background of the participants (as a gfO'J?) £or the topic of this 
Institute was: 

very adequate 12 3 4 5 very inadequate 

(1) (2) . 

5t As a group, motivation and interest of the participants appeared to be: 

very high 12 3 4 5 very low 

(2) (1) 

6. In terms of following your (or the director's) instructions and schedule, 
participants were: 

very cooperative 12 3 4 5 very uncooperative 

(3) 

7. The next Traveling Institute on this topic should be: 

considerably longer 12 3 4 5 considerably shorter 

(3) 

8, For this group of participants, the content presented was: 

too easy I 2 3 4 5 too difficult 

(2) (1) 

too theoretical 12 3 4 5 too non-theoretical 

(3) 

very useful 12 3 4 5 useless 

(1) (1) (1) 

9, Participants' knowledge and/or skills related zo this topic have: 

increased considerably 12 3 4 5 shown no incrense 

(1) (2) 
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10. As a facilitotor 1.: :naKjnci arrnu;. o.rn«ints tor i.iiis ln:;lxuu: • Jac ciforUs 
of AKIiA Central Offica vtnit wdin-: 

vc>ry helpful J 2 3 4 5 not at nil helpful 
(1) (2) 

11. Forroative evaluation feedback proved: 

very helpful 1 2 3 4 5 not at ail helpful 

(2) (1) 

12. As a result of formative evaluation feedback you made: 
many changes I 2 3 A 5 no changes 

(1) 0) 0) 

13. Overall, the evaluation procedures v/ere: 

not at all disruptive 1 2 4 5 very disruptive 

(1) (2) 

14. Overall, your objectives for thir- Institute were: 

attained 12 3 4 5 act attained 
(2) 

(Please be as specific as possible in responding to the open-c-ided questions l/;*.lcw. ) 

15. Now that the Institute is over, what things do you feel went c-specially we2iV 



16. What areas do you feel are most in need of xraprovemcnt before ne:<t time? 



REPORT OF LONG-TERM EFFECTS OF AERA TRAVELING INSTITUTES 
AS PERCEIVED BY INSTITUTE PARTICIPANTS 
(December, 1973) 



Marilyn Averill 



Laboratory of Educational Research 
University of Colorado 



A questionnaire concerned with possible long-term benefits obtained 
from traveling institutes was sent to 138 people, (A copy of the 
questionnaire is in Appendix A.) This group consisted of all partici- 
pants at the first six traveling institutes (2 Bayesian Statistics, 3 
Alternative Conceptions of Evaluation, 1 Performance Based Teacher 
Education). This report summarizes responses to this questionnaire. 

Returns were received from 96 respondents; of these, three referred 
to later institutes (the respondent having attended more than one 
traveling institute) and were not included in the data analysis. Four 
more questionnaires were returned as undeliverable, and three were 
returned after all analyses had been completed. Due to time constraints 
and the high rate of returns (70%), there was no follow-up mailing. 

Data Analyses 

All analyses were based on the data from 86 questionnaires (62% of 
teial) and included frequency counts, means, standard deviations, and 
contingency tables. 

The questionnaire can be divided into two sections: 

1. Items recording demographic data, and 

2. items concerned with professional grov/th resulting from attendance 
at the Institute. 



Description of the Respondents 

Questions were included concerning employing institutions (Item 1) 
primary professional role (Item 2), percentage of professional time 
spent in various activities (Item 3), and AERA divisional membership 
(Item 4), Frequency counts of responses to Items 1, 2, and 4jand means 
for Item 3 are included in Appendix A. 

A majority (53, or 62%) of the respondents are employed in colleges 
or universities; no other employing institution exceeds a frequency of 
7 (8%). The most frequent professional role was that of teacher (32^ or 
37%), followed by evaluator (16>or 19%), researcher (14^ or 16%), and 
administrator (12, or 14%). Portions of professional time spent in 
various activities followed a similar pattern, with the greateist amount 
of time spent in instruction (X = 31%), AERA divisional memberships 
v/ere spread across all divisions, but the largest number of respondents 
(52) were members of Division D, Measurement and Research Methodology ♦ 

Evaluation Results 

Items 5, 6, 7, and 8 question the extent to which the respondent 
feels he has increased professional proficiencies as a result of the 
institutejor has been able to use the content of the institute in his 
professional activities. Means and standard deviations of responses to 
this item are presented in Table 2. 

Responses to these items were moderately negative with means varying 
from 3.90, concerning use of the content of the Institute in the prepar- 
ation of professional papers, to 3.06, concerning the increment in 



professional skills resulting from institute attendance. Negative 
responses seem to occur heavily in a few institutes; the specific 
institute varies across questions, although Performance Based Teacher 
Education appears consistently to draw more negative responses than 
either of the other two topics. 

Question 9 asks whether the respondent would now choose again to 
attend the institute. A majority (54^ or 63%) indicated that they would 
again choose to attend, 19 (22%) said they would not, and 10 (12%) were 
undecided. 

Contingency tables were constructed to examine the relationship 
between institute attended and responses to questions 5 through 9. 
These tables appear in Appendix B. 

Only fourteen people responded to the open-ended question asking 
for other useful skills and knowledge acquired at the institute they 
attended. The only response occurring more than once was given by 10 
participants, who stated that although they have not yet had an oppor- 
tunity to use their new skills, they do feel that the institute gave 
them knowledge and skills which they may be able to apply at a later 
time. 

Summary 

. 

Participant responses to the first six institutes were in general 
slightly negative. It appears that results would be somewhat more 
positive if the data from the Performance Based Teacher Education 
institute were excluded from the analysis. However, 63% of all respon- 
dents indicated "^hat, if given the opportunity, they v/ould again choose 
to participate in the traveling institute. 



Appendix A 

Questionnaire, with item response frequncies, means and standard deviations. 
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amenican eoucaTionaL 
ReseaRCH associarion 



Dear Participant: 

The AERA Research Training Committee is evaluating the 1972-73 series of 
Traveling Institutes. Part of this evaluation involves a follow-up for all 
Institute participants to survey your long-term opinions about the Institute. 

We would appreciate it if you would take a few moments to complete and 
return the following questionnaire. 



Evaluation of AERA Traveling Institute 

Pi recti ons : Please read each question carefully and be completely candid in 
your responses. Your name will not be associated with your responses in any 
way. 



Name of the Institute you attended: 

Location of the Institute you attended: 

1. You^ primary employment is presently in a: {check ONE) 

,5-^ College or University 

2^ Public School System 

^ Educational R&D Center 

3 Regional Educational Laboratory 

3 Federal Government 

V State Department of Education 
s I am presently a student 

Other (please specify) , 

3 'B/a^t: 

2. Your primary role at your employing institution is that of: (Check the ONE 
tha^ is most applicable) 

^-^A Teacher 

Administrator 
Researcher 
Evaluator 
A Developer 

l_ Supervisor 

Student 

V Other (please specify) 

3. What portion or your professional time do you spend in each of the following 
activities :^/)1fi4./t 5) 

Instruction ^ij} % Administration % Student (formally enrolled) 3. 
Development or Dissemination ^ % Research or Evaluation 3L> % 
Other ( ) 



1 SUId'NfM STREEir. N W • WASHINGION. D C 2003b • 202'223-9485 

ERJC 




2 



4. Please check the AERA Division(s) of which you are presently a member: 
(check ANY that apply) 

10 A (Administration 

^ B (Curriculum and Objectives) 

C (Learning and Instruction) 

^ji D (Measurement and Research ftethodology) 

^ E (Counseling and Human Development 

l_ F (History and Historiography) 

^ G (Social Context of Education) 

H (School Evaluation and Program Development) 

I I am a student member of AERA 

/X 0 I am not presently a member of AERA 

5- To what extent do you feel that attendance at the Institute has increased 
your professional skills? 

to a great extent 12 3 4 5 not at all 

6. To what extent have you utilized the knowledge you acquired at the Research 
Training Institute you attended? 

to a great extent 12 3 4 5 not at all 

7. How professionally valuable would you rate the contacts you made at the 
Institute? 

very valuable 12 3 4 5 worthless 

8. How much has the ccfitent of the Institute aided you in: 

a. the preparation of professional papers? 

very much 1 2 3 4 5 not at all 
f = 3.9c Sy - /.JLf 

b. the conduct of professional research? 

verv much 1 2 3 4 5 not at all 
)t'3.3^ Sy - f.3/ 

c. the preparation of courses taught? 

very much 1 2 3 4 5 not at all 

On the back of this page, please list other areas in which you feel skills or 
knowledge acquired at the Institute have been useful to you. 

9. If you were able to choose over again, would you have attended this Institute? 
5V Yes No fo Uncertain 3 iBlawtC 



THANK YOU FOR YOUR COOPERATION 
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Table 1 

Institute attended vs, responses to Item 5, *7o what extent do you feel that 
attendance at the Institute has increased your professional skills? 



To a great extent 

1 


2 


3 


4 


Not at all 

5 




Bayesian 
(Amherst) 


0 


3 


H 

(22) 


(33) 


/ 




Bayesian 
(Tampa) 
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7 

W/) 


3 

in) 


3 


1 
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Evaluation 
(Portl and) 


0 


3 


(, 

mo) 


// 


0 

(0) 


IB 

iho) 


Evaluation 
(Tampa) 




6 

C?q) 


6 


6" 


0 

ic) 


J-6 

C/oc) 


Evaluation 
(Tucson) 


I 

Oo) 


3 

C3o) 




O 

Co) 


V 




Teacher Education 
(Boston) 


0 


4) 


is) 


(.33) 


Csq) 


(<{<) 



Row percentages appear in parentheses. 
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Institute attended vs. responses to Item 6, "To what extent have you utilized 
the knowledge you acquired at the Research Training Institute you attended?" 



To a great extent 

1 


z 


3 


4 


Not at all 

5 




Bayesian 
vflmnerst j 


o 

it) 




1 




3 




Bayesian 
(Tampa) 


f 




r 








Evaluation 
(Portland) 


0 




*i 






t/Ol) 


Evaluation 
(Tampa) 


(p 


s 




f/ 


0 




Evaluation 
(Tucson) 


0 


3 


3 


U) 




/o 


Teacher Education 
(Boston) 


1 


0 

to) 


0 

Cc) 


(0 

t5€) 


V. 


// 



Row percentages appear in parentheses. 
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Table 3 

Institute attended vs. responses to Item 7, "How professionally valuable would 
you rate the contacts you made at the Institute?" 



Very valuable 

1 


2 


3 


4 


Worthless 
5 




Bayesian 
v/vnnersx; 


/ 

C$) 


/ 


J. 

Ul) 


tso) 




/A 

Odo) 
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(Tampa) 


/ 

u) 
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0 
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3 


C3i) 




1 

a) 


/3 
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3 
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£L 


It 
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a 


1 

Uo) 


a. 


3 


A 


16 


Teacher Education 
(Boston) 


1 

(J) 


e> 








i:l 



Row percentages appear in parentheses. 



Table 4 

Institute attended vs. responses to Item 8a » "How much has the content of the 
Institute aided you in the preparation of professional papers?" 



Very much 

1 


2 


3 


4 


Not at all 

5 


roh.1 


Bayesian 
(Amherst) 




1 


1 

a) 
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(Tampa) 


1 


3 

Oi) 








n 
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0 


2 


/ 


/ 
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13 


Evaluation 
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f 
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ao 
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I 

00 


0 
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o 
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Teacher Education 
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o 
CO 


1 
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0 
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^3 



Row percentages appear in parentheses* 



Table 5 



Institute attended vs. responses to Item 8b, "How much has the content of the 
Institute aided you in the conduct of professional research?" 



Very much 

1 


2 


3 


4 


Not at all 

5 
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0 
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3 


3 
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ao 
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1 
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,3 
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0 
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V 
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1 


3 
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3 


3- 
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Row percentages are in parentheses. 



Table 6 

Institute attended vs. responses to Item 8c, "How much has the content of the 
Institute aided you in the preparation of courses taught?" 



Very much * Not at all 
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4 
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Row percentages are in parentheses. 




Table 7 



Institute attended vs. responses to Item 9, "If you were able to choose over 
again, would you have attended this Institute." 





Yes 


No 


Uncertain 
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(55) 
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Row percentages appear in parentheses. 



Evaluation of AERA Traveling Training Institute 
"Research in Performance-Based Teacher Education" 



Directed by 



Frederick McDonald 
hducational Testing Service 



May 2-4, 1973 
Boston, Massachusetts 



-1- 



The first Traveling Training Institute on "Research in Performance Based 
Teacher Education" was conducted in Boston, Massachusetts, on May 2, 3 and 4, 1973. 
Twenty-five panticipants attended the three day institute directed by Frederick 
McDonald, ETS, New Jersey, with John Krumboltz, Marlene Katz and David Potter as 
instructional staff members . 

Four participant instruments were administred pre and/or post according to 
the evaluation design employed for previous research training institutes- The 
cognitive and affective instr\iments were initially drafted by AERA's central office y 
the final version of the instrximents wab the result of changes made by the instruc- 
tional staff. The most significant modification in the instruments produced a 
cognitive essay test that required staff scoring. Due to the cancellation of the 
session (dicussed later) these instruments were not scored and therefore the cogni- 
tive gain scores are not available for this institute. In addition^ the cancellation 
obviously precluded the use of the evaluation report in a formative sense to improve 
subsequent presentation of the Institute. Therefore, this report is presented in an 
abbreviated form. 

Participant reactions to item numbers 6, 12, 19, 24, and 28 in tables 3 & 4 
highlight the level of dissatisfaction in this training institute. An analysis of 
these and related questions- coupled with the generally negative responses to open- 
ended questions provided ample evidence for the Association's Standing Committee on 
Research Training to unanimously agree to withdraw sponsorship of subsequent 
occurrences of this Institute. 

In the opinion of AERA's designated on-site evaluator, several circumstances 
contributed to the demise of this institute. An overriding level of frustration was 
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generated by the gap between participant expectations and what the Institute was 
able to deliver. This was compounded by the heavy emphasis, (approximately 70-80% 
of the structured class time) in small group discussions and activities. Instruc- 
tors sitting in these sessions generally played the role of an observer, which re- 
sulted in participants spending time sharing their knowledge and experience among 
themselves. The lack of closure to these small group sessions and the limited 
opportunity for questions to be directed to the instructional staff, contributed 
to a heightened level of frustration. 

The fact that the director of the institute was called home to attend to a 
personal emergency for one day of the work shop and the invited guest lecturer 
misunderstood the day he was to appear and therefore did not attend, further handi- 
capped the success of this training institute. 

The staff of this institute planned a detailed schedule of activities after 
giving consideration to participant input from the application form. However, upon 
the advice of the on-site evaluator and a hastily-devised participant reaction form, 
a variety of changes was instituted. Most notably was an atteitpt to include more 
content-orientated lectures. However, this resulted in a great amount of repetition 
among the speakers. It became obvious that in spite of the staff's willingness to 
institute changes, they were not in a position to meet the varied expectations and 
objectives of the participants. 

The tables which follow present the relevent data available from this Institute. 
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Table I 
Description of Participants 



Employing Institution 



College or 
University 

14 



Regional Cents c 
or Lab* 



Federal or State Dept. 
of Education 



Student Other 
2 3 



Teacher 
9 



Primary Professional Role 



Administrator 
5 



Resecurcher 
2 



Evaluator or 
Developer 



Student 
2 



Other 
1 



Academic Degrees 



Master's 
5 



Doctorate 
17 



(B) Curriculum & 
Objectives 

8 

(E) Counseling & Human 
Development 



Membership in Divisions of AERA 

(C) Learning & Instruc- 
tion 

10 



(D) .Measurement & Research 
Methodology 

6 



(G) Social Context of 
Education 



{ ) Not presently a 
member of AERA 
0 



( ) Student 
1 



Reasons for Attending the Institute 



Desire to gain new skills 
or knowledge related to 
the topic. 



Desire to improve 
existing skills or 
knowledge related to 
the topic . 



Desire to discuss 
problems related 
to the topic with 
experts. 



8 



11 
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NO 

Experience 
0 



Prior Experience 



Limited 
Experi'*:nce 

. 15 



Considerable 
Experience 

8 



Great Amount 
Experience 

1 



less than 
50 miles 



Distance of home from Institute site 



50-100 
miles 



100-500 
miles 



500-1,000 
miles 



more than 
1000 miles 
3 
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Table 2 



Tabulation of Participant Responses to Critique 
Items Relating to Pre-Institute Planning 



Item X 

1- How important dp you feel the topic of this 
Institute is to educational research and/or 
evaluation? 

very important 1 2 3 4 5 very unimportant 1.35 
(17) (5) (0) (1) (0) 

2. How important do you feel the topic of this 
Institute is to educational practice? 

very iit^ortant 1 2 3 4 5 very unimportant 1.30 
(17) (5) (1) (0) (0) 

4. Leaving aside the quality of instruction for 
the moment do you think the topic treated in 
this Institute shoijild be treated again in future 
Institutes? 

definitely 1 2 3 4 5 definitely not 1.41 

(14) (7) (1) (0) (0) 

18. In your opinion, the staff members were in 
general : 

very well-qualified 1 2 3 4 5 very unqualified 3.45 

(2) (2) (6) (8) (4) 

23. The meeting room facilities for the Institute 
were: 

excellent 1 2 3 4 5 poor 3.09 

(2) (4) (9) (6) (2) 

25. Considering what you have learned by attending 
this Institute (or any other benefits you have 
received from attendance), the tuition fee was: 

considerably too low 1 2 3 4 5 considerably 4.43 

(0) (0) (3) (7) (13) too high 



Note: Frequencies are shown in parentheses. 
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Table 3 

Tabulation of Participant Responses to Items 
on Conduct of the Institute 



Item 



How relevant was the content of the Institute to the 
topic that was advertised? 

highly relevant 1 2 3 4 5 not at all 3.61 
(1) (3) (7) (5) (7) relevant 

The staff's objectives for this Institute were: 

very clear to 1 2 3 4 5 very unclear 4.00 
me (1) (2) (3) (7) (10) to me 



7 No, they were ambiguous 



4.52 



6. The planning and organization of this Institute was: 

excellent 1 2 3 4 5 poor 

(0) (1) (3) (2) (17) 

7. Overall, was the Institute long enough to cover the 
topic adequately? 

considerably 1 2 3 4 5 considerably 3.36 

too long (0) (0) (16) (4) (2) too short 

8. As a rule, daily sessions were: 

considerably 1 2 3 4 5 considerably 3.05 

too long (0) (3) (16) (2) (1) too short 

9. Do you feel all necessary prerequisites were clearly stated 

in information you received prior to the Institute? (check ONE) 

5 Yes, they were clearly stated 



11 No, prerequisites were listed 

10. Did you receive advance reading materials from the Institute 

director (s) early enough to read them prior to the Institute? (check ONE) 

1 Y es, materials came early enough 



0 No, materials came too late 



22 No materials were sent 



11. Do you think you entered the Institute with the appropriate 
prerequisites or prior knowledge to make what you learned 
there of value to you? 2.71 

Q 1 had more than 1 2 3 4 5 1 was seriously lack- 
enough prepara- (15) (Jy) CO) (1) i2\ ing in preparation 
mamam tion 1.70 



(Table 3 cont,) 
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Item 

12 • In teintis of your background and preparation for 
the Institute, the content of this Institute was; 



excellent 12 3 4 5 
(13) (3) (6) (0) (0) 



poor 



1.68 



13. Overall, the quality of instruction in this 
Institute was: 



excellent 1 2 3 4 5 poor 
(1) (0) (5) C7) (10) 



4.08 



14. The instruction was generally: 



too lecture-oriented 



12 3 4 5 too discussion- 
(1) (3) (5) (6) (6) oriented 



3.62 



very interesting 



1 2 3 4 5 very uninteres- 
(0) (1) (9) (7) (5) ting 



3.72 



very informative 



1 2 3 4 5 very uninfontia- 
(0) (1) (6) (7) (9) tive 



4.04 



15. Opportunities for asking questions were: 



sufficient 



1 

(7) 



2 

(6) 



3 

(4) 



4 

(4) 



5 

(2) 



insufficient 



2.48 



opportunities for studying were: 



sufficient 



1 

(2) 



2 

(0) 



3 

(4) 



4 

(9) 



5 

(6) 



insufficient 



3.81 



17. The amount of work the staff required of you was: 



far too much 



1 

(1) 



2 

(2) 



3 4 
(9) (5) 



5 

(5) 



far too little 



3,50 



18. Opportunities for you to interact with the staff with 
respect to problems in your own work which relate to 
the topic were: 



sufficient 



12 3 4 5 insufficient 
(1) (2) (7) (3) (10) 



3.83 



19. * In your opinion, the staff members were in general: 



very well- 
prepared 



1 2 3 4 5 very un- 
(2) (2) (6) (8) (4) prepared 



3.45 
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Item 



20 • Did the staff seek your reactions to their instructional 
procedures, scheduling, etc, during the Institute? 

frequently 1 2 3 4 5 never 2,91 
(3) (3) (11) (5) (1) 

21, Did it appear to you that your reactions led to improvement? 

frequently 1 2 3 4 5 never 3.65 
(1) (3) (4) (10) (5) 

22. The formal evaluation of this Institute by the "outside 
evaluation" (e.g., those instruments) was: 

not at all dis- 1 2 3 4 5 very dis- 2.00 
ruptive (11) (2) (6) (1) (1) ruptive 



Note: Frequencies are shown in parentheses. 
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Table 4 

Tabulation of Participaiit Responses to Items on the 
General Evaluation of the Institute 



Item X 

24 • Do you anticipate that what you have learned from 
the Institute will be useful to you in your own work? 

definitely 1 2 3 4 5 definitely not 3.36 
(2) (2) (7) (8) (3) 



28* If you were cible to choose over again, would you attend 
this Institute? 

(3) Yes (18) No (2) Uncertain 



29. If this Institute were held again, would you recommend to 
a colleague that they attend? 

(4) Yes (16) No (3) Uncertain 



Note: Frequencies are shown in parentheses 
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EVALUA^^ION OF AKi^A TnAVlLLiriG INSTITUT 
"Altemat.ive Conceptjons of Evaluation 

Berkeley, Ca] If omia 
July 197'3 



Robert M. Stonehill 



laboratory of Educational Research 
University of Colorado 



Thi:: ntport. Ir, an evaluation of the : cu:'ih "Al terriative Conceptiorio 
of r. valuation*' Travolin^'; Tnstltutf^^ pTCsoni': 1 [;y uaniel .-ituf'i lebear^j Kichael 
::criven an;i Dian^- R-dnharri In Berkeleyi C- 'ifomia on July 5 and 6, 1973« 
Descriptions of the evaluation design and :yC ^^eneral Institute organization 
arc available in detail in the precedinp^ evaluation reports of this series, 

Inr.tltute Participantr. 

Seventeen participants attended the Institute, three of whom left 
early and did not fill out final evaluation forms, and one of whom filled 
out only Participant Instrument #1 (the participant characteristic information 
instrument). Nine participants represented institutes of hip:her education! 
although all categories except State Department of Education were represented, 
AdministTHtor wa:: the most prevalent T)rimary role, with only one researcher 
and two evaluators represented. Twelye of the participants held Doctorate 
decrees, vrhile sixteen of the seventeen held decrees above a B.A. Sixteen 
of the seventeen have had previous evalioative experience, with ten reporting 
considerable or a p:reat amount of experience with evaluation. 

Ar in all previous Institutes, many participants travelled great distances 
to attend; ten participants travelled more than 1000 miles to attend, and only 
one person (a graduate student at Berkeley) came from the Bay Area. This 
seems to indicate, as with previous Traveling Institutes, that th'5 convenience 
of a date seems more important than the location of the Institute, 

See Appendix, 1 for a copy of Participant Instrument #1, along with a 
frequency count of participant responses. 

Conduct of t he Ini'>ti t ute 

The first meeting of the Institute vias scheduled for 8i00 Thursday morning 
but did not begin until 9t30. During this time the preliminary evaluation in- 
struments were distributedt everyone completed Participant Instrument #1, while 
a quasi -randomly selected half filled out Participant Instruments #2 and #3 
(self-rr?port of perceived knowledge, the cognitive instrument and the affective 
instrument, respectively). Only one person had not received advance materials 
and this was dvie to late registration for the Institute on his part. A planning 
session had taken place the night before among the Institute's directors, and a 
r^isource table had been set up and the agenda finalized. 

Lists were secured from the participants of their definition of evaluation, 
their favorite evaluation problems, and their justification for using evaluation. 
These were to be commented on at some later time by the directors. 

y -e Appendix 2, the ASRA Training Institute Tentative Agenda, >rtilch was 
fHaaMSffld fairly closely, with the exception of the evening session of individual 
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rearliriicr and. us*? of the reference table, 

Scriven spoke first on Pathway Svalualinn (though several particimnts 
lid not reco^niz.e 1.t as such as indicated by thei^- responses on Participant 
Instrument ^ and Rtufflebeam foil owed v^^th a dlsc^ission of the ^ipp model. 
There w^^.v^ some complaints that written hr: ■■vlo^it--- wouV, .greatly facilitate the 
presentation, as a Tot of time vias s'oent i '\ copying dom information presented 
by the directors. The situation was somevr.iat rectified the next day when 
:)tufflebeam had XEROXed copies made of hi$. presentation for people to follow along 
with rather than to scribble down. 

The meeting adjourned for lunch at 1?.:30, and reconvened at 2tl5 with a 
presentation by Diane Reinhard on advocate teams. Scriven followed with a 
discutsi.ion on Goal-free evaluation* Stuff lebeam sroke briefly about evaluation 
problems. At 4: SO the Institute ended for the day, with about 1^ people lasting 
through the entire first day's meeting. 

Thirteen people were present at 9s 00 on F*riday morning, though the Institute 
didn't start until 9t^5 with Scriven' s presentation of the product checklist. 
There was no use ^^f handouts at this time* At 11:40 Stuff leb'='am lead a dis- 
cussion of evaluation prob?ioriS posed the previous day by the participants. 
Lunch was from 12:30 until 1:40, at which time the directors all discussed 
participants' individual evaluation problems. Between ten and twelve participants 
attended the final session. The Institue ended at 3^00, and the final evaluation 
of the Institute ended at 3*50. 

Evaluation Results 

The design paradigm followed in evaluating the Berkeley Institute was 
identical to that used in the Portland and Tucson evaluations, and will not 
be discussed af?:ain here. 

An analysis of variance performed between the two posttest groups (pretested 
and unprotested) yielded no significant difference between groups either on the 
affective (Participant Instrument #3) or the cognitive (Participant Instrument 
ffZ) instrument. Note that a higher r.core is desirable on the affective inr;trument, 
while a lower score is desirable on the cognitive instriiment. This should be kept 
in mind while examining the ANOVA tables with regard to group means. (See Appendix 4) 

An analysis of variance performed between the pretest and the posttest 
for the unpretested group also yielded no significant difference, either on the 
affective or cognitive measures. 

A repeated measures design for analysis of variance was also performed 
Q a pretest - posttest comparison on the same person. There was a significant 
^ll^Ference between the pre- and posttests on the cognitive instrument, with 



ANALYSIS (Vi-' VARIANCE TABLES 



-.CURCK 
OF 

/ARIA7T0N 
WITHIN 



[Am 

SCiUAHFJS 
370.8571 



PRETEST (AKf'eCTIVE, PRE- 
TESTED GROUP) X 
P0STTS5T (AFPEGTIVa, NON- 
PRETESTED GROUP) 



TOFAL 389.7500 15 

PRETEST GROUP: n = 9, X 
NGN -PRETEST GROUP: n = 7, X 



^^-6.^7 
ij-S. 36 



F = .7132 



SOURCE 
OF 

VARIATION 



SUM OP 
SQUARES 



HE/J-I 
DP SQUARE 



BETWEEN 



WITHIN 



89.2857 



851.71^3 



14 



f'9.2H57 



60. H 367 



PRETEST (COGNITIVE, PRE' 
TESTED GROUP) x 
POSTTEST (COGNITIVE, NON- 
PRETESTED GROUP) 



TOTAL 9^1.0000 15 

PRETEST GROUP: n - 9, X = 3^.3 
NON -PRETEST GROUP t n = 7, X = 29.6 



F = 1.4676 



SOURCE 
OK 

VARFA" TON 
BET'./EEN 



WTTl^IN 
TOTAL 



SUM OF 
SQUARES 

12.0714 

332.8571 
344.9286 

PRETEST GROUP: n 
NON-PRETEST GROUP: n 



DP 



1 



12 ! 

I 

I 

13 I 

7, ^ = ^7.0 
7, X = 48.9 



MEAN 
SQUARE 

12.0714 

27.7381 



PosTTEST (affj;ctive, 

PRETESTED GROUP) x 
POSTTEST (affective, 

NON -PRETESTED GROUP) 



F = .4352 



.-JOUHCE 
OF 

VARIATION 



?.|',TWEK!.' 
WT'l'HTN 



SUM OF 
SQUARES 

16.0714 
459.'J2P-6 



MEAN 
DP SQUARE 

1 16.0714 
12 ' 38.2R57 



POSTTEST (COGNITIVE, 

PRETESTED GROUP) : 

POSTTEST (COGNITIVE, 

NON -PRETESTED GROUP) 



TOTAL 



475.5000 



13 



PRETEST GROUP: n = 7, X = 27.4 
NON -PRETEST GROUP: n =■ 7, X = 29.6 



F = .4198 



ANALYSTS OF VARIANCi-J ^'m ViiFh-ATSD MEASURES 
uible 1. Pretest vs. Por.ttest (prete:3tea f/roup only) -Affective instrument 



sourgf: of variation 



OF 
SQUARSS 



DP 



MEAN 
SQUARE 



S ( SUBJECTS ) 



M (MD'.ASURt'JS) 



345.0000 



3.5000 



57.5000 



3.5000 



8.85 



.5^ 



3M (PJHROR) 



39.000c 



6.5000 



Preter,t: n = 7, X = 46.0 
Posttest-; n = 7, X = 47.0 



Table 2. Pretest vs. Posttest (pretested ^^roup only) -Cognitive instrument 



SOURCF: Oh' VARIATION 



SUM OF 
SQUARES 



DF 



MEAN 
SQUARE 



S (SUBJECTS) 



882.8571 



147.1429 15.39 



M (MlilASURES) 



77.7857 



77.7857 8.09 



SM (MRROR) 



57.71^3 



9.6190 



Pretest t n = 7, X =■ 32.1 
Posttest:n = 7, X = 27.^ 



p^.05 
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the Tiostt^^r-; t oxhibitlm-^ a better percept*) r.n the participant's knowled^z:e of 
ovaiuation. This war-, si p:nil"iaant at the .')^ le^'el. There was no significant 
difference between pre- and -posttest on tin) affective instmoment in the re- 
peated inea:.mres analysis. In both cases, there was a significant difference 
(beyond the ,01 l<'Vol) among subjects in the repeated measures design. 

Responses to the Cxitique form were ^!:enerally much more positive than (#4) 
.those for the Tucson Institute « Frequencies of responses, as well as appropriate 
means and standard deviations appear in Appendix 3« 

In general, -participants felt the st^iff merabers were very vjell-quaiif led 
ana v^ry we 11 -pre pa red. The topic was held to be an impoTtant one to both 
educational research and to educational practice. 

Total expenses for the Institute ranged from $0 to $600, with a mean of 
$3^4-8. Personal expenr^es (those not covcr-.d by the empioyinK institution) 
ranged from $0 to $475f with a mean of $116. This Is considerably hiF.hcr 
than the reported personal expenses for the Tucson Institute, and people 
p;eneTally felt that the tuition fee was too high. 
Conduct of the Institute 

As with the other Institutes, there is a wide variety of responses within 
cate.c^ories, but the evaluation in general is superior to that of the Tucson 
Institute. The planning and organisation were rated good, and the content of 
the Institute was rated as relevant to the topic advertised. The quality of 
instruction was rated very good. Opportunities for asking questions and to 
interact viith the staff wore rated sufficient. However, most neople felt that 
the amount of work required of the participant was too little. Ten people 
said they would attend the Institute if they had to choose over, while only 
one said he would not. Daily sessions were found to be the right length, x^hile 
pc^ople felt the Institute was too short to cover the topic adequately. 
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• v:\v\rivy and ciecomTr.encia^tions 

The evaiuation in f?;eneTcil v/as quite i^vorable, much more so than that of the 
'uc^'.cn Institute. Mor.t of the surrctJiotion;, .la^ie by the Tucivon evaluation were 
.iTiplemented for the Berkeioy Institute, aiajm/, therr.s Ali :^:taff memberp. arrived 
•>n t-.ime and stayed until the concluoion of the Institute, all but one participant 
•rnceived their advance materials, and a hc:.vy emphasis was placed on discussion. 

There was still some criticism of the title of the Institute from those 
Participants who expected more of a debate fonnate between Scriven and Stufflebeam. 
It seems to this evaluator that any stated objective., of the Institute are 
secondary (or tertiary), since most of the participants he spoke with were primarily 
Interested in listoninr: to Scriven and Stufflebeam talk and interact with one another 
■ ind uLth the participants, Uo one the evaluator spoke to was seriously dis~ 
appointed in this respect. 

More work should be expected of the participants. 

Much of the lecture content can be handed out as advance material before the 
Lecture to avoid the predilection with copying rather than comprehending material. 

Participants should be told in advance that they may iiave an opportunity 
to present evaluation problems for the consideration of the group, and thus be 
better prepared to offer examples for discussion, 

Summary of ODen-'cnded Questions 

■\nditional -^rtn^cts of the Institute which v;ere of value: 
Dircussion of product evaluation 

Opfinness of staff to contact with participants after Institute is over 
Two days forced attention to evaluation Droblems 

'llements missing from Institute which would be of future value in Institutes of 
this t;rr-c: 

More targeted on major evaluation issues or topics 

More case studies and perhaps evaluation simulation or game 

Problem solving sessions 

elements you would suggest sacrificing from present format to make room for 
now torics: 

General background and descriptive material that should have been disseminated 

prior to Institute 
Long presentation for specific instances 
Theoretical discussions brought forth by participants 
Free time other tb^n informal short breaks 

Critici^;mr. or suggestions concerning time allocation for Inr-.titute activities: 

Mot enoup:h time to treat any topic in depth 
Too mvch product evaluation 

More on actual techniques and less on history of participants 
Q Kovc time on individual problems, varied format 



JcJjoduie r.hould rave nore va^'lcty 

C'onveniont, av^a.y Vro]n oiTlce 

Ot)t)ort.unities for In-depth study 
Choice of da tor. locations 

Second cb^ance to attend one dependin^^, on feedback received firoin. earlier Inistituti 
Small ^roup dynamics and interactions 

Time to dif-est information before experiencing overload of information like at 
AERA 

Major dir-.advanta/?:e?-. of traveling institutes: 

Cof:t is hi.^h for special trips 
Time not sufficient 
Tim.e awa-y from .tob 
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Topic oi tiio liuil. ll^ilP yi)u ,n*<.« .U t <Mui i u>; : 

Alternative Conceptions of Evaluation 



Location of the Inj^titutG ,you aro nLtending (state): 

Berkeley, Calif. 



You, are presfintly eiiipjoycci in a: (check the ONE tliat is most applicable) 

Coliege or University (9) 

Public School System (2) 

Educational R&D Center (1) 

Regional Laboratory (?) 

Federal Govornment (2) 

State Department of Education 

r am presently a student (1) 

Other (please specify) 

Your primary roie at your employing institution is that of: (check the 
ONE that ig most a?)plicabie) 

t eat'her (3) 
administrator (7) 
researcher (1) 
eva 1 uator (2) 
devpioper (3) 

supervisor 

s t uden t (1) 

other (please specify) 

UhaL academic de^^-^es do yon hold? (check the most advanced degree held) 

bachelor's (1) 

mar. ter 9 

doctorate (12) 
other (please specify) ^ 

I'lease check the Division(s) of AERA of which you are presently a member. 

A (Administration) 

B (Curriculum and Objectives ) (5) 

C (Learrr and Instruction) (3) 

D (Measurement and Research Methodology) (?) 

iL (Counselinj^ and Human Development) (2) 

F (History and Historiography) 

G (Social Context of Education) (2) 

U (School Evaluation and Program Development) (5) 

Student Member 

1 am not presently a member of AERA (?) 



7. iiuv did you rtr«t. loarn about thi: ,:it: i:.ute? (check ONE) 

• profeGttionn] pubi i.c^Li on (pi.ase specify) Sd ucational Researcher (9) 

col I(iag'.iC? oi r.r.udont (^^) 

mailed aotfrt- . , 

other (please ::pocify) \'^[ 

8, What are your primary reaiions tor attending thi3 session? (check ONK) 

desire to gain new skills or cvnovledge related to the topic (6) 

desire to l.nprove existinjr. li ills or knowledge related to the topic (.5) 

desire to discuss problenu; r.; lated to the topic with experts (6) 

other (please specify) 



9. Wliat prior experience have you had with the topic of this Institute? 
(cncck one) 

no experience whatsoever (l) 

have had limited experience (e.p,., have retjd c: lit tle about the topic) (6) 

have had considerable experience (e.g., took a class on the topic, (7) 

or worked for several months on activities where the topic was used) 
great amount of experience (c.^;., have used the topic for several (3) 

years, or have taken several classes on the topic) 

10« If this session were not available now, hue was offered instead as a session 
for a i^Ainilar auount of time either immediateiy before or after tiie 
AliRA convention, would you choose to attend the session there? 

(Assume registration fees, etc., remained constant) 

11. 1^^^^ y^^^ fit tend the AEKA Convention in Nev; Orle ans this year? 
(6) Y es (10) No 



li« Is 'che date selected for the Institute convenient for you? 
(I5>es (2) No 



13. Is the site selected for the Institute convenient for you? 
(12) Yes (5) Ho 



1^ . Mow far is your home from the site of this Institute? (check ONE ) 

( ^ ) less than 30 miles 
"T^T^ 5U-iOO miles 

U) 101-500 miles 

(2) ' 501-1000 miles 
TTTOT] m^vre than J ,000 miles 



\PPKWI)IX 2 



July S • (j , 107 3 
Berkeley , I^alifornia 



i>, 1973 



8:30 - 9:30 



9:30 -11:30 



Koqis tratiorj - evaluation 



< PuuhwcAy Evaluation Problems # 
CI I'P Kodel i;i5Cussion 



Diane Remhard 

Michael Cicriven 
Daniel StuLflcbeam 



11:30 - 1:00 
1:00 - 3:00 



Lunch 

Advocate Team Technique 
Goal-Free Evaluation Discussion 



Diane Reinhard 
Michael Scriven 



3:00 - 4:30 



Introduction to Jim Sanders' 
Material on Evaluation Problems 



Michael Scriven 
Daniel Stufflebeam 



•;:30 - 5:00 
5:00 - 6:00 



Construcc questions to hand in 
Individual Reading Cuse of 
reference table) 



Diane Reinhard 



6:00 - 6:30 
8:30 -10:00. 



Optional Group Dinner ? 

Individual Reading ( use of 
reference table) 



Diane Reinhard 



July 6, 1973 
9:00 - 11:00 



Special Topic Presentations 
based on questions 



Michael Scriven 
Daniel Stufflebecon 



Discussion 



11:00 - 12:00 



Participants ' Individual 
Problems 



Michael Scriven 
Daniel Stuff lebt/m 



12:00 - 



1:00 



1:00 
2:00 



Lunch 

Participants • Individual 
Problems 



Michael Scriven 
Daniel Stufflebecim 



2:00 - 3:30 



Wrap-up and General Discussion 



Michael Scriven 
Daniel Stufflebeam 



3: 30 



4:00 



Final Evaluation 



Robert Stonehill 



AERA Traveling Institutes 
Participant Instrument #4 

ji recti ons : This evaluation form is administered by the AERA Research Trainino 
-onmittee or persons assisting them wUh this evaluation. Completed forms wilt 
oe returned directly to the Corrmittee by the person administering this form. 
After the data have bren tabulated, the instructional staff of your session 
•nay request a sunmary. The principal purpose of this form 1s to assist In the 
nlannlng of subsequent traveling Institutes. Therefore, be completely candid 
in your responses. Do not sign your name. 

Topic Of this Institute ai.ter!;ativ:-: couc t '.mons of ev aluatjon 
Location of this Institute BERKELEY, CALIF. 



f-"or each question or partial statement below, please circle the number which best 
reflects your reaction. 

X = 1.14 I. How Important do you feel the topic of this Institute is 'to educational research 
s = .16 and/or evaluation? 

very important 1 | 3 4 i very unimportant 

How important do you feel the topic of this Institute is to educational practice? 

very Important 12 3 4 5 very unimportant 

(10 3 1 ) 

How relevant was the content of the Institute to the topic that was advertised? 

highly relevant 1 2 3 4 5 not at all relevant 

(5^41) 

Leaving aside the quality of instruction for the moment, do you think 
s - .61 the t opic treated in this Institute should be treated again in future Institutes? 

definitely 12 3 4 5 definitely not 

(11 2 1 ) 

X = 2.91 ). The staff's objectives for this Institute were: 
3 = 1.07 

very dear to me 1 2 3 4 5 very unclear to me 

(1 ^ 5 3 1 ) 
X = The planning and organization of this Institute was: 

excellent 12 3 4 5 poor 
(17 5 1) 
X = 1.50 Overall, was the Institute long enough to cover the topic adequately? 

considerably too long 12 3 4 5 considerably too short 

( 1 8 2 3 ) 



X = I.IS 2. 
s = .63 



X = 2.07 3. 
s = 1.00 



X = 1.2Q 



»= ''..00 3. As a rule, dail^ sessions were: 



f= .7;' 
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considerably too long 12 3 4 5 considerably too short 

(11 9 3 ) 



Do you ffiel all necessary prerequisite, wore clearly stated in information you 
received prior to the Institute? (chetik ONE) 

(10) Yes, they were clearly stated 

' nTy"" Nn, they wore ambiguous 

T.1 ) "N o prerequisites were listed 

Did you receive advance reading materi:;ls from the Institute director(s) early 
enough to read them prior to the Institute? (check ONE) 

(13) Yes, materials came early enough 
No, materials came too late 
No materials were sent 



X = 



X 
s 



X = 

X = 

X = 

s ~ 

X = 

r, = 

X = 

s 

X = 



2.6: 
.6^ 



2.0V 



Do you think you entered the Institute with the appropriate prerequisites or prior 
knowledge to make what you learned the.^e of value to you? 



I had more than 



1 



5 

1 ) 



I was seriously 
lacking in preparation 



2, 



enough preparation ^ ^ 

In terms of your background and prepo)^itio^ for the Institute, the 
content of this Institute was : 



too 

elementary 



1 



too 

advanced 



3/ 



(661 ) 
Overall, the quality of instruction in this Institute was: 



excellent 1 2 3 
( ^ 5 5 
The Instruction was generally: 



) 



poor 



.n? 



l.V 



too lecture-oriented 
very interesting 
very informative 



1 

( 

1 

( 8 
1 

( ^ 



2 

6 



3 

11 



3 

3 



4 
4 

4 

1 



too discussion-oriented 
very uninteresting 
very uninformati ve 



Opportunities for asking questions were: 

sufficient 12 3 4 5 

( 12 2 
Opportunities for studying were; 



57. 



sufficient 12 3 4 5 

{ k ^ 3 2 1 ) 
The amount of work the staff required of you was: 



insufficient 



insufficient 



far too much 



1 



3 4 5 
( 2 9 3) 

Opportunities for you to interact with the staff with respect to problems In 
your own work which relate to the topic were: 



far too little 



18. 



X = 



1.2? 



sufficient 



1 

( 3 



2 

5 



3 

2 



4 

3 



5 

1) 



insufficient 
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In your op-;n^on, the staff memfcers wc.-'? in qeneral : 



o.; very well -qua! 1 f1 ed 1 ? 3 1 5 very unqualified 

( n 1 ) 

very wel l-prepored 12 3 4 5 very unprepared 

(7 5 1 1 ) 



1 .?! 

'..1 



r- = .91 



Did the staff seek your reactions to Vieir instructional procedures, schtduling. 
etc.. duri ncj the Institute? 

f'-equentiy 12 3 4 5 never 

( 6 5 2 1 ) 

Did it apceor to you that your reacticis led to improvements in tho ins true tiorui 
procedures, scheduling, etc.? 

X ~ 2.7C froquently 12 3 4 5 never 



X = 1.1- 



The formal evaluation of this Institu^j by the "outside evaludtion" 
(e.g., those instruments) was: 

nut at an disruptive ,1 2 3 4 5 , very disruptivo 

( 11 2 ) • 



il. The meeting room facilities for the Institute were: 

^. = 1 1/. excellent 1 2 3 4 5 poor 

( 6 5 2 1) 

t *. Do you anticipate that what you have learned from the histitute will 
be useful to you in your own work? 

X = 1-7^ definitely 12 3 4 5 definitely not 
^""^ (6611 ) 

Con'sidering what you have learned by attending this Institute (or any other 
benefits you have received from attendance), the tuition fee was: 

X = 7 64 

, ^ ''pa considerably too low 1 2 3 4 5 considerably too hinh 

( 1 5 6 2 ) 

.5 Approximately how much money (including travel costs, living expensf^s , 
X = $llb 07 tuition) fiid this Institute cost you personally (i.e., was not 

- <tico'oo reimbursed)? $ 
r» — J. jc. » ( } 



What was the total amount of money (including travel, living expenses, 
•t^^-'.o/ arid tuition) your attendance cost, including personal costs to you an^d 
$190.56 costs paid by your institution or other sources? ^$ 

r'3 If ycu were able to choose over again, would you attend this Institute? 
^^!i].jes il^No Uncertain 

If this Institute were held again, would you recommend to a f.olleaque that 
he attend? 



(10) Yes (1) No (3) 



Uncertain 
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30. How would you rate Che value of each 
(Piease clrclti ONE for EACH row) 



X=?.l'- a. Scr'ven: pathway evaluation 
"-t.il' 

X=l.Wi ^J_ Stuff lebeam: profckvns, C!PP 
moael 

X^=1.21 Reinhard: advocate team 

3=1*31 technique 

X=1.6'i- • Scriven; Cjoal-fre^ evaluation 

e. Workbook examples 



f '.he following aspects of this Institutf? 



X=3,00 •\ 
s=l .00 

X=1.7S 
s- .67 

s=1.01 
)(=l.o'-, i. 

s= .69 
X=2.7a j 



individual reading: use of reference 
table 

"Special topics" presentation 

Discussion of individuel problems 

General question and answer 
sessi ens wi th staff 

I'lfornial discussion with peers 



no opDC tuni ty 
to jud g e 



NA 

{ (' 

NA 

( 3 

NA 

( 3 

NA 

( 3 

NA 

( 17 ) 

NA 

( 6 

NA 



very 
valuable 



( 8 

NA 
( 3 



NA 
{ ^ 
NA 

{ 3 



1 


2 

3 


3 

2 


4 

2 


) 


1 
1 

!i- 




2 




' ) 


1 

2 


2 
2 


1 

3 


5 


2 ; 


I 

6 


'J 

r. 
7 


i 
1 


/J 


) 


] 


i. 


'\ 
f 


1 




1 


7 

h. 


3 

ii. 


A 

<\ 

^. 


1 \ 


1 

3 


7 
5 


3 
1 


'1 


■ ) 


1 

2 


f 


3 
5 


4 

3 


'') 


1 


2 
7 


3 

2 


/I 


) 


1 

2 


'J 
c 

5 


3 
2 


4 





Pl<}ai.e list an> asoects of the Institute other than those listod in if) ahnvc 
which vou felt w(^re of considerable value. 



Pledi'; list any criticisms or suggestions you have concerning time ano(.atiori 
\rv any o^" the Institute activities or sessions. 



a. Please list any elements which were missing from this Institute that you 
feel would be of value in future Institutes of this type. 



D. What A"0U:d you suggest sacrificing from the present Institute format to 
i'lake room for new topics or activities? 
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S. Jay Samuels 
University of Minnesota 
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The evaluation of Model B's training session involved the admin- 
istration of four instruments three for the participants and one for 
the staff members. These instruments were revised forms of the instru- 
ments used in the evaluation of the traveling training institutes. 
Copies of the instruments used can be found in the appendix. Formative 
evaluation was the responsibility of the session staff; but Mrs. Gruenberg, 
who administered the instruments, assisted with this task. 

Participant Instrument ifl was administered to those participants 
present just before their first session meeting began. Participation 
Instrument #2 was administered as a pre-test to a random half of the 
participants at the same time as Instrument //I. Participant Instruiuent 
#3, the Staff Instrument, and Participant Instrument #2 (as a post-test) 
were all administered near the close of the last meeting of each session. 
It was intended that all participants complete Instrument //2 as a post- 
test, with those who had been pre-tested indicating this on their post- 
test. The plan failed in that only a very few post-tests that were 
marked to indicate that the individual has been pre-tested. There was 
also a problem in general getting the instruments returned. Mailed 
follow-ups, where attempted and possible, also met with poor response. 

The survey of AERA. members at large showed that although only 31 percent 
of the respondents attended the annual meeting ( and two percent attended 
an AERA pre- or postsession) , 83 percent were aware of the pre- and 
postsessions. The major reasons for not attending an^^ sessions were 
scheduling problems in which participants could not be in (or remain in) 
New Orleans during the time the course was offered. 
Q Participant Instrument #1 is tabulated below. Generally the exact 

ERIC 

™"ffl"ext of the item is not included in this tabulation. 
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Participant Instrument #2 results are tabulated below. Included 
with each item tabulation is the item mean and standard deviation. These 
values were obtained by assigning responses values from 1 to 5, with 
S corresponding to what would be thought to be the most positive res- 
ponse. The 5 corresponds with responses nearest the right-hand side 
of the page unless the item is preceded by a capital R, in which case 
the order is reversed and the 5 response is nearest the left-hand side 
of the page. 

The tabulations are ordered with the pre-test conu.ng first then 
the post-test. Each post-test is followed by analysis of variance 
table. Total scores for each returned pre-test and post-test were 
analyzed. No P ratio approached significance at the .10 level. 
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AERA Pre- session 

Research on Improving Decoding and Reading Comprehension 

Pre Test 
Participant Instruiivent #2 

Directions: This opinionnaire attempts to assess attitudes of participants toward 
educational evaluation. Please indicate your agreement or disagreement with each 
statement by checking ( ) the answer that best describes how you personally feel, 
regardless of whether other people may agree or disagree with you. Although many 
of the questions may appear similar, please judge each one on an individual basis. 
Since we need to know your attitude, please answer each question frankly and honestly. 
There are no correct responses. There is no time limit, but respond as quickly as 
you can, and do not leave out any of the statements . 



1, I see and feel deeply a need for increased infoinnation cJDout reading 
research for decision-making in my field. 

Strongly Disagree Undecided or Agree Strongly 
Disagree Neutral Agree 

(0) (0) (0) (5) (3) 

2, Reading research usually results in arbitrary judgments about the 
educative process. 

Strongly Disagree Undecided or Agree Strongly 
Disagree Neutral Agree 

(0) (3) (2) (3) (0) 

3, There is currently too much concern with decoding in reading education. 

Strongly Disagree Undecided or Agree Strongly 
Disagree Neutral Agree 

(0) (2) (2) (3) (1) 

4, Educators do not receive sufficient training in reading comprehension. 

Strongly Disagree Undecided or Agree Strongly 
Disagree Neutral Agree 

(0) (1) (2) (3) (2) 

5, Reading processes are too conplex to be studied objectively. 

Strongly Disagree Undecided or Agree Strongly 
Disagree Neutral Agree 



Mean 
4,38 

S,D 
,52 



Mean 
3,00 

S,D 
,93 



(2) 



(4) 



(1) 



(1) 



(0) 



Mean 
2.63 

S.D 
1,06 



Mean 
3,75 

S,D 
1,04 



Mean 
3.88 

S.D 
.99 
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6. Decoding in reading is an atteit^t to reduce education to a mechanistic 

process . Mean 

3.38 

Strongly Disagree Undecided or Agree Strongly S.D. 

Disagree Neutral Agree 1.06 

(1) (3) (2) (2) (0) 

7. Reading research usually results in improvements in educational practice* 

Mean 

Strongly Disagree Undecided or Agree Strongly 2.50 

Disagree Neutral Agree S.D. 

1.07 

(1) (4) (1) (2) (0) 

R 8. Intuition and general knowledge of practitioners are more valuable than 

reading research in making decisions in education. Mean 

3.88 

Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree S.D. 

.35 

(0) (7) (1) (0) (0) 

9. Using decoding and reading con^rehension research information in my work 

does not appeal to me. Mean 

4.25 

Strongly Disagree Undecided or Agree Strongly S.D. 

Disagree Neutral Agree .46 

(2) (6) (0) (0) (0) 

10. Money spent on research contributes more to the inprovement of reading 

education than any other expenditure. Mean 

2.63 

Strongly Disagree Undecided or Agree Strongly 

Disagree Nexitral Agree S.D. 

.92 

{0) (5) (1) (2) (0) 
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AERA Pre-Session 

Research in In5)roving Decoding and Reading Comprehension 

Post Test 

Participant Instrument #2 

Directions: This opinionnaire atternpts to assess attitudes of participemts toward 
educational evaluation. Please indicate your agreement or disagreement with each 
statement by checking ( ) the answer that best describes how you personally feel, 
regardless of whether other people may agree or disagree with you. Although many 
of the questions may appear similar, please judge each one on an individual basis. 
Since we need to know your attitude, please answer each question frankly and honestly. 
There are no correct responses. There is no time limit, but respond as qtiickly as 
you can, and do not leave out any of the statements. 

1. I see and feel deeply a need for increased information about reading 

research for decision-making in my field. Mean 



Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree 

(1) (0) (0) (5) (6) 



4.25 

S.D. 
1.14 



^ 2. Reading research usually results in arbitrary judgments about the 

educative process. Mean 

3.33 

Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree S.D. 

1.23 

(2) (4) (3) (2) (1) 

1^ 3. There is currently too much concern with decoding in reading education. 

Mean 

Strongly Disagree Undecided or Agree Strongly 2.33 

Disagree Neutral Agree 

S.D. 

(0) (2) (2) (6) (2) .98 

4. Educators do not receive sufficient training in reading con^rehension. 

Mean 

Strongly Disagree Undecided or Agree Strongly 3.08 

Disagree Neutral Agree 

S.D. 

(2) (1) (3) (6) (0) 1.16 



ERIC 



5. Reading processes are too complex to be studied objectively. 

Mean 

Strongly Disagree Undecided or Agree Strongly 3.75 

Disagree Neutral Agree 

S.D. 

9^ C3) (5) (3) (0) (1) 1.14 



-11- 



6. Decoding in reading is an attempt to reduce education to a mechanistic 
process . 



(3) (5) (0) (3) (1) 

7. Reading research usually results in improvements in educational practice. 

Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree 

(3) (6) (2) (1) (0) 



ERIC 



Mean 
3.50 



Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree S.D. 



1.38 



Mean 
2.08 

S.D. 
.90 



p 8. Intuition and general knowledge of practitioners are more valuable than 

reading research in making decisions in education. Mean 

3.83 

Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree S.D. 

1.03 

(2) (8) (1) (0) (1) 

9. Using decoding and reading comprehension resesirch information in my work 

does not appeal to me. i Mean 

4.42 

Strongly Disagree Undecided or Agree Strongly 

Disagree Neutral Agree S.D. 

.52 

(5) (7) (0) (0) (0) 

10. Money spent on research contributes more to the improvement of readinc 

education than any other expenditure. Mean 

2.25 

Strongly 

Disagxee Disagree Undecided or Agree Strongly S.D. 

Neutral Agree 1.06 
(3) (5) (2) (2) (0) 
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Analysis of Variance for Reading Research Pre-Post 
Attitude Scale (Participant Instrument #2) 



TREATMENT GROUP 1 2 

SAMPLE SIZE 8 12 

MEAN 34.250 32.833 

STANDARD DEVIATION 4 . 921 3 . 040 



SUM OF SQUARES 
BETWEEN GROUPS 9.6333 
WITHIN GROUPS 271.1667 



ANALYSIS OF VARIANCE 

DF MEAN SQUARE 

1 9.6333 
18 15.0648 



F. Ratio 
.6395 



Total 



280.8000 



19 
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The tabulation of Participant Instrument #3 follows, Teibulations 
are similar to those of Instrument #1^ except the text of the item is 
usually included for tliis tabulation. Meeuis and standard deviations 
as well as maximvuns and minimxims appear where appropriate • The res*-* 
ponses to the open ended items are as close to verbatim as possible. 
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1973 AERA Pre and Post Training Sessions 

Staff Instrument #1 

Topic of Session Reading Research 

As part of our evaluation of the AERA Training Sessions, we feel it 
iit^ortant to obtain your reactions, as directors or staff members, to several 
aspects of the session. We will appreciate your candor in responding. 

For each partial statement printed below, please circle the number which 
best reflects your reaction to that statement. 

1. The meeting rooms were: 

excellent 1 2 3 4 5 poor 
(0) (0) (1) (2) (0) 

2. The living 

excellent 1 2 3 4 5 poor 
(2) (0) (1) (0) (0) 

3. The number of participants was: 

too large 1 2 3 4 5 too small 
(0) (0) (2) (0) (1) 

4. The background of the participants (as a group) for the topic of this 
session was: 

very adequate 1 2 3 4 5 very inadequate 

(0) (2) (1) (0) (0) 

5. As a group, motivation and interest of the participants appeared to be: 

very high 1 2 3 4 5 very low 
(2) (1) (0) (0) (0) 

6. In terms of following your (or the Director's) instructions and schedule, 
participants were: 

very cooperative 12 3 4 5 very uncooperative 

(3) (0) (0) (0) (0) 

7. The next Training Session on this topic should be: 

considerably longer 12 3 4 5 considerably shorter 

(0) (0) (1) (0) (2) 
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8. For this group of participants, the content presented was: 

too easy 1 2 3 4 5 too difficult 
(0) (0) (3) (0) (0) 

too theoretical 1 2 3 4 5 too non-theoretical 
(0) (0) (3) (0) (0) 

very useful 12 3 4 5 useless 
. (1) (0) (2) (0) (0) 

9. Participants* knowledge and/or skills related to this topic have: 

increased considerably 1 2 3 4 5 shown no increase 

CD (2) (0) (0) (0) 

10. As a facilitator in making arrangements for this Institute, the efforts 
of AERA Central Office staff were: 

very helpful 1 2 3 4 5 not at all helpful 
(2) (0) (0) (0) (0) 

11. Formative evaluation feedback proved: 

very helpful 1 2 3 4 5 not at all helpful 
(2) (0) (0) (0) (0) 

12. As a result of formative evaluation feedback you made: 

many changes 1 2 3 4 5 no changes 
(1) (0) (1) (0) (0) 

13. Overall, the evaluation procedures were: 

not at all disruptive 1 2 3 4 5 very disruptive 

(1) (0) (1) (0) (0) 

14. Overall, your objectives for this Institute were: 

attained 1 2 3 4 5 not attained 
(2) (0) (1) (0) (0) 

(Please be as specific as possible in responding to the open-ended questions below) 

15. Now that the Institute is over, what things do you feel went especially well? 

1. Excellent exposure of staff points of view, which differed widely. 

2. Individual relations between staff and participants. 

3. Very provocative discussion, both among staff and between staff and 
participants. Good coverage of Important theoretical issues. 
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What Areas do you feel are most in need of improvement before 
next time? 



1. Some suggested prior readings for participants. Mote form 
structure in program - or at least an opportunity for staff 
to make a program coherent. 

Any other comments on ways to improve the sessions or the precedures 
for evaluating the sessions? 

1. Formative evaluation is important. We found it helful, 

2. Make clearer to participants and instructors the purpose of rAe 
Institute. 
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I. Research on Reading Acquisition: With An Emphasis 
on Deprived Populations 

Date: February 21 • 25 

Cost: Five days, AERA members— $225; 

nonmembers — S250 
Director: S. Jay Samuels, University of Minnesota 
Instructors: Jack Bormuth, University of Chicago 

Frank Smith, Ontario Institute for 
Studies in Education, Universit; of 
Toronto 

Joanna Williams, Tachers College, 

Columbia University 
Ken Goodman, Wayne State University 

This session will discuss a number of unresolved issues 
such as the role of language variation and its interference in 
reading, existence of subskills in reading and the role of in- 
telligence differences in reading acquisition. The faculty rep- 
resents different viewpoints on these problems. The objectives 
consist of presenting data on each of these issues and intro- 
ducing new unpublished information on how to facilitate 
decoding and comprehension. These new approaches to de- 
coding and comprehension are of special relevance for our 
poor population. 

This session is designed to train a broader audience than 
just academic research producers. This session will include 
such topics as: (a) language differences among the disadvan- 
taged; (b) special curriculum needs of the learning-disabled 
child; (c) operationalization of the comprehension process 
for instruction; (d) twelve easy ways to make reading diffi- 
cult; (e) new ways to facilitate perceptual learning; (f) role 
of intelligence in reading, and how to minimize the effects of 
low I.Q. in reading acquisition; (g) theories of the reading 
process for beginning and skilled readers; and (h) prereq- 
uisites for reading. 

The session is suitable for curriculum supervisors, teachers 
of learning-disabled students, personnel who work with 
inner-city residents and the disadvantaged, school evaluators, 
curriculum designers, reading specialists, and researchers in 
verbal learning. 

The' content of the session will be presented so thai par- 
ticipants will be able to comprehend all major concepts. Pro- 
visions have been made to include blocks of time for faculty- 
audience discussion. Evening discussions will be participant- 
organized and run with staff taking a part, but not a leading 
role. 

No special training or background beyond that usually 
found at professional meetings is required. Selected materials 
will be given to the audience. Additional materials, such as 
books, will be available for temporary loan. 



11. Computer Managed Instruction 

Date: February 23, 24, 25 / 

Cost: Three days, AERA members -S 135; 

nonmembers — SI50 
Director: Frank B. Baker, University of Wisconsin 
Instructors: Jack McManus, Southwest Regional 

Laboratory 



Robert Berger. Southwest Regional 

Laboratory 
George Behr, University of Wisconsin 

One of the important recent developments in education has 
been the emergence of computer-based instructional man- 
agement systems (CM I). Such systems represent a new mode 
of computer involvement in the instructional process and 
they will assume an increasingly important role in future edu- 
cational research. Because CM! systems are quite new and 
little systematic literatare exists, it is difficult for educational 
researchers to become familiar with the field. Consequently, 
the primary objectives of this session is to provide the par- 
ticipants with a concise, up to date, examination of the field 
of CML This examination will focus upon the conceptual 
basis of CM!, the design rationale of existing CM! systems, 
and the role of CM I systema as a powerful vehicle for con- 
ducting a wide range of educational research. 

Upon completion of the session, participants will have 
acquired an understanding of the role played by CMI systems 
in the instructional process, will be aware of the *'state of the 
art'' in the field of CMI; and establish a relationship between 
their own research and CML 

This session is designed to meet the needs of two rather 
distinct groups. One is the educational researcher whose 
work could be facilitated by becoming involved in CML 
Second are persons in the public schools who need to ac- 
quire a working knowledge of CMI in order to participate 
in CMI related research. Although the actual involvement 
of these two groups in CMI could be quite different, they 
have a common need for an exposure to the most recent 
developments in the field of CMI. These two audiences will 
complement each other in the discussion groups. Content 
will be presented in a building block fashion that presumes 
no prior knowledge of CMI. The concepts underlying CMI 
will be presented before actual CMI systems are examined. 
In this way the participants will be brought to a common 
background level. Through the combined use of lectures, 
small and large group discussions, audio-visual presentations 
and computer demonstrations, participants will be shown 
the many facets of CMI quickly and effectively. Small discus- 
sion groups will enable persons with common interests to 
work together at a level that is consistent with their abilities. 
"Hands on" experience will be provided that will enable the 
participants to actually use a CMI system via a computer 
terminal. 

It should be noted that it is assumed the participants have 
no prior computer experience. A course outline, copies of 

By papers, system documentation, and instructional ma- 
s will be provided participants prior to the training 
)n. 
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RESEARCH ON READING ACQUISITION, WITH AN EMPHASIS ON 
DEPRIVED POPULATIONS 

S. Jay Samuels, University of Minnesota 
Joanna Williams, Columbia University 
David LaBerge, University of Minnesota 

This Traveling Training Institute will discuss a number of unresolved issues 
such as the role of language variation and its interference in reading, existence 
of subskills in reading and the role of intelligence differences in reading ac- 
quisition. The faculty represents different viewpoints on these problems. The 
objectives consist of presenting data on each of these issues and introducing 
new unpublished information on how to facilitate decoding and comprehension. These 
new approaches to decoding and comprehension are of special relevance for our poor 
population. 

This session is designed to train a broader audience than just academic re- 
search producers. This session will include such topics as: (a) language differ- 
ences among the disadvantaged; (b) special curriculum needs of the learning-dis- 
abled child; (c) operationalization of the comprehension process for instruction; 
(d) twelve easy ways to make reading difficult; (e) new ways to facilitate per- 
ceptual learning; (f) role of intelligence in reading, and how to minimize the 
effects of low I.Q. in reading acquisition; (g) theories of the reading process for 
beginning and skilled readers; and (h) prerequisites for reading. 

The session is suitable for curriculum supervisors, teachers of learning 
disabled students, personnel who work with ix*ner-city residents and the disadvan- 
taged, school evaluators, curriculum desigr.ers, reading specialists, and researchers 
in verbal learning. 

The content of the session will be presented so that participants will be able 
to comprehend all major concepts. Provisions have been made to include blocks of 
time for faculty-audience discussion. An evening discussion will be participant 
organized and run with staff taking a part, but not a leading role. 

No special training or background beyond that usually found at professional 
meetings is required. Selected materials will be given to the audience. Addition- 
al materials, such as books, will be available for temporary loan. 

ic -k -k ic 

The first offering of the Institute will be on June 18 and 19, 1973 in New 
York City. The second presentation of the session will occur in Minnesota during 
August. The registration fee for the Institute is $90 for AERA member, $100 for 
non-members . 

Requests for applications and inquiries should be directed to the Central 
Office of AERA. 

AMERICAN EDUCATIONAL RESEARCH ASSOCIATION 
1126 Sixteenth Street, N.W., Washington, D.C. 20036 
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RESEARCH TRAINING INSTITUTES 



The Amerioan Educational Research Association will sponsor a number of training sessions 
during 73^74. The Institutes are carefully selected^ designed and organized to meet some 
of the specific training needs of educational researchers and practitioners. The Jnsti-' 
tutes listed below indicate the initial occurance. Some of these sessions will be re^ 
peated in different regions of the county. For abstracts of these sessions^ applications^ 
future sites and announcements of additional Training Institutes write: Research Training^ 
AERA Central Office^ 1126 Sixteenth Street j N.W.j Washington^ B.C. 20036 or consult issues 
of the Educational Researcher. 



Title: Research in Performance Based Teacher Education 

Director: Frederick J. McDonald^ ETS; Hew Jersey 

Instructors: Bruce R. Joyce^ Columbia University 

John D. Krumboltz^ ETSi Hew Jersey 

Marlaine Lockheed KatZj ETS; New Jersey 

David A. Potter^ ETS; New Jersey 
Date: Three day 8^ May 2^3^ 4 

Cost: AERA members - $l35j Non-^members $150 

Location: Boston^ Massachusetts 

****************************** ********** 



Title: 

Director: 

Instructors: 

Date: 
Cost: 
Location 



Criterion Referenced Measurement and Instructional Improvement 

Eva L. Baker J University of California^ Los Angeles 
Jason MiZlman^ Cornell University 
Evan KeisloTj University of California^ Los Angeles 
Two days J September 
AERA members - $90^ Non-^members $100 
San Diego^ California 



****************************************************************************************** 



Title: Research on Reading Acquisition, With an Emphasis on Deprived Populations 

Director: 5. Jay Samuels^ University of Minnesota 

Instructors: Joanna Williams^ Columbia University 

David LaBerge^ University of Minnesota 
Date': Two daysj June 

Cost: AERA members - $90^ Non~members $100 

Location: New York City 

******************************************************************************************* 

Title: Theory and Methodology of Research of Written Instruction 

Directors: Ernst F. Rothkopf and Laurence T* Frase^ Belt Telephone Lab.j New Jersey 

Date: Three day 8^ August 3j4j5 

Cost: AERA members - $135 Non-members $150 

Location: Madison Wisconsin 

******************************************************************************************* 

Title: Alternative Conceptions of Evaluation 

Directors: Michael Sariven^ University of California^ Berkeley 
Daniel Stufflebeam 

kins true tor: Diane Reinhaxd 

Date: Two day s^ July 5^6 

Cost: AERA members - $90^ Non-members $100 

Location: Berkeley^ California 

Q t**********************************************--V************************************** 
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Call for Proposals: 1973 
Research Training Sessions 

Proposals arc now being solicilcd 
for the 1973 Research Training 
Sessions lo We held in conjuclion 
with (lie AliRA Annual Meeting In 
New Orleans, February 26— March 
K 1973 (Mardi Grus Commences 
March 3rd). 

As in previous years, these ses- 
sions are designed to upgrade the 
research competencies of individ- 
uals engaged in educational re- 
. search. In addition lo ^htse how 
traditional training sessions, an 
OlTicc of luhicalion grant lias been 
awarded lo AliRA for conducting 
two new pre- or post-sessions spe- 
ciricaiiy planned lo train a much 
broader audience than (he narrowly 
defined educational research pro- 
ducer. These Iwo sessions will be 
designed for audiences such as de- 
velopers and evaluators in schools, 
academic researchers in disciplines 
such as anthropology, political sci- 
ence and economics, and for those 
involved in educational research 
with minorities and with urban en- 
vironment. 

Individuals interested in pro- 
posing and directing an AERA 
Training Session should write to 
(he Research Training Chairman, 
Frank Hi Furlcy, for a proposal 
outline. The sessions will vary in 
. Icngih two to five days dc- 

j^^ on the nature of the train- 
■ irig to be conducted. There are no 
restrictions on content; it is hoped 
that a broad range of topics will be 
proposed. To continue present pol- 
icy of making training sessions a 
self-supporting activity, partici- 
pants will be charged a fee. 

The deadline for submission of 
proposals is July 12, 1972, Reviews 
and recommendations on proposals 
will be made by representatives of 
• the Divisions. Final selection will be 
the responsibility of the Standing 
Committee on Research Training. 
Prospective directors will be noti- 
flc4 in early August of the accept- 
ability of their proposals. 
Q *lcqucsts for proposa] outlines 



and inquiries should be addressed 
to: l-rank M. F'arley, Wisconsin 
Research and Development Center 
for Cognitive Learning, University 
of Wisconsin, 1404 Regent Street, 
Madison, Wisconsin, 53706. 

Mini-Course Instructors 

In an effort lo explore ways in 
which the Annual Mcetmg. can be 
used to improve the research com- 

* petencies of its member;>, AERA 
will sponsor two- to four-hour 
mini-courses during the 1973 An- 
nual Meeting in New Orleans. The 
purpose of (he sessions wifl be to 
transmit sprcific .skills in research, 
development, and evaluation to . 

. participants rather than to carry on 
a general discussion. Mini-course 
instructors will receive a modest 
honorarium and/or travel expen- 
ses; participants will be expected to 
pay a small fee. 

Individuals interested in serving 
as instructors should prepare a 
short statement specifying what 
the participants will be able to do 
by virtue of attending the instruc- 
tional session. The intended skills 
to be developed should be identi- 
fied in as specific terms as possible. 

By July 15. 1972, mail this state- 
ment and a self-addressed envelope 
to Jason Millman, Stone Hall, Cor- 
nell University, Ithaca, NY 14850. 

Experimental Conversations 
Planned for '73 Meeting 

In an effort to explore ways in 
which the Annual Meeting can be 
used to improve the research of its 
members, AERA will experiment 
with facilitating conversations be- 
tween established and less exper- 
ienced educational researchers 
working in common areas of inter- 
est. These conversations will take 
place during the 1973 Annual Meet- 
ing in New Orleans. Here is how 
they will work. 

1. If you are pursuing a line of 
research inquiry you would like dis- 
cussed by a prominent researcher, 
nominate the individual and in one 
to three sentences describe your 
area of research interest* 



2. From among the more fre- 
quently mentioned nominees, in- 
dividuals will be invited to lead 
conversations. 

3. If your nominee is selected 
and agrees to participate, then 
shortly before the Annual Meeting 
you may be invited to prepare a one 
lo two page statement describing 
your research ideas and any ques- 
tions you would like to see discus* 
sed. 

4. This statement will be sent to 
the individuals nominated with the 
hope that it will inllucnce the con^ 
tent of the conversations. 

Attendance at t^^sc conversa- 
tions will be controlled by a ticket 
system similar to that employed 
with the graduate student seminars. 
Unlike the graduate student semi- 
nars, however, these conversations 
are open to all conference partici- 
pants and the discussion should be. 
focused on specific concerns of the 
nominators rather than on general 
issues. 

Research Conversation 
Nomination Form 



Name . 



Institution . 



Street . 



City/State/Zip . 



Your Nominee 



His/Her Institution . 



Line of Research YOU are pursu- 
ing: : _ 



ERIC 



May 1072 



This form and a stamped, self- 
addressed envelope should be mail- 
ed by July IS to Jason Millman, 
Stone Hall, Cornell University, 
Ilhaca, NY 14830, 
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CaU for Proposals 

1974 Research Training Program 

Proposal are now being solicited 
lor AHRA's 1974 Research Train- 
ing Program. As in previous years, 
the training sessions are designed 
to upgrade the research competen- 
cies of individuals engaged in educa- 
tion research and development as 
well as for an audience broader than 
the narroNviy defined educational 
research producer. There are no 
restrictions on course content, and it 
is hoped thai a wide range of topics 
will be proposed. 

Proposals for training sessions in 
1974 will be considered for any of 
the three modes listed below: 

1. Pre or Post Sessions — These 
training sessions will vary in length 
from two to five days depending on 
the nature of the training to be con- 
ducted. The sessions will be held in 
conjunction with the 1974 annual 
meeting in Chicago, April 15-19. 

2. Training Institutes — These two 
to five day institutes will be con- 
ducted one or more times during 
the year in different regions of the 
country. 

3. Mini Training Courses — Four 
hour training sessions to transmit 
specific skills in R, D & E will be 
held during the 1974 annual meeting 
in Chicago, April 15-19. 

To enable the research training 
program to maintain a self-support- 
ing status, participant c will be 
charged a registration fee. How- 
ever, to establish these fees at a min- 
imal rate, directors are strongly 
urged not to impose an arbitrary 
limit on enrollment. 

The deadline for submission of 
proposals is August I, 1973. Reviews 
and recommendations on proposals 
will be made by representatives of 
Divisions. Final selection is the re- 
sponsibility of the Standing Com- 
mittee on Research Training. Pro- 
spective directors will be notified in 
September of the acceptability of 
their proposals. 

Requests for proposal outlines 
and inquiries should be addressed 
to: Research Training Committee, 
AERA, 1 126 Sixteenth Street, N.W., 
Washington. D.C. 20036. 
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MINI RESEARCH TRAINING COURSES 



during 



The Annual Meeting 



Attached are the descriptions and location of four 3 or 4 honr 
training sessions that will be conducted during the annual meetir.; ^ 
These sessions are designed to transmit specific skills in educa~ 
tional research, development and evaluation. 

Participants should register and obtain a ticket at the regis- 
tration area of the MarriGtL or Jung Hotels. Early registration is 
encouraged as attendance will be limited to 30 participants per 
session. 

A registration fee of $8.00 will be assessed each participant. 
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\TITLE : 



Computers in Educational Research 



DESCRIPTION: This is a computer literacy course that will enable participants 
to generalize about, and discriminate between, computer systems, 
options and languages and make intelligent decisions about their 
uses. Participants will be able to: 

1, (a) identify the basic components of computer systems and 
(b) describe their functions 

2, (a) identify optional components of computer systems 

(b) describe their functions and 

(c) rate their importance for research applications, 

3, Write short programs in a current programming language 
and using this language as a model, be able to: 

(a) describe the basic components of computer languages 
[ (b) describe the techniques usually used to implement them 

(c) identify the hardware components necessary for the 
implementation and 

(d) rate languages regarding their appropriateness for 
research applications* 

4, Describe a number of "program packages" that are commonly 
used in educational research, 

INSTRUCTORS: Ronald G. Ragsdale and Sorel Reismau 

INSTITUTION: Ontario Institute for Studies in Education 

4 hours 



LENGTH: 

TIME & 
PLACE: 



Tuesday, February 27, 1973, 2:15-6:15 
Chartres, Marriott Hotel 



TITLE: 



DESCRIPTION: 



Nonparametric Analysis of Variance on Main Effects and Interaction 
Based on Normal Scores 

Often behavioral researchers discard plans to perform a classical 
t or F test in ANOVA designs because of assumption violation. 
Until recently, little could be suggested to help a researcher if 
the designs contained nested or crossed factors, but now highly 
efficient alternatives have appeared based on normal scores. Some 
of these methods are presented in this minicourse. Statistical 
tests ) planned and post hoc methods of analysis based on normal 
scores are presented for one, two, nested and crossed designs. 
Attendants will be taught how to use these methods with confid- 
ence and explain the results of a normal scores analysis to lay 
readers or listeners. Instruction will be conducted from exten- 
sive and complete handouts. 



INSTRUCTOR: Leonard A. Marascuilo 

INSTITUTION: University of California at Berkeley 



LENGTH : 
ERJC: 



3 hours 

Wednesday, February 28, 1973, 2:15-6:15 
Mardi Gras F, Marriott Hotel 



TITLE: 

DESCRIPTION: 



INSTRUCTORS: 

INSTITUTION: 

LENGTH: 

TIME & : 
PLACE: 



Planning Formative Evaluations for Instructional Products 

The purpose of formative evalui^tions is to determine which com- 
ponents of an instructional product need to be improved. This 
training session will (1) introduce the participant to the major 
considerations involved in planning a formative evaluation — 
statement of purpose, selection of measures, sample, design, 
method of analysis, criteria for decisions to revise, administra- 
tive plan, and (2) provide the participant with experience in re- 
viewing and criticizing summaries of formative evaluation plans. 

The training session will be conducted primarily as a series of 
problem/simulated situations in which the participants as mem- 
bers of Smctll teams, will be asked to design u.^ redesign formative 
evaluation plans. In addition, there will be general presentations 
on the structure of and criteria for assessing formative evaluation 
plans. 

The training session is particularly designed for persons interested 
in or responsible for the development or revision of instructional 
materials and programs. Knowledge of elementary descriptive statis- 
tics is assumed. 

Eva L. Baker, Director; Evan Keisler, Peter Leung, Merlin C. Wittrock 
and Edys Quellmalz 

University of California at Los Angeles and SWRL 
3 hours 



Monday, February 26 1973, 2:15-5:15 
Meeting Room 2, Jung Hotel 



TITLE: 

DESCRIPTION: 



INSTRUCTOR: 

INSTITUTION: 

LENGTH: 

TIME & : 
PLACE : 



Sample-Free Item Calibration and Test-Free Person Measurement 

Participants will learn how to use the log odds response model for 
the expression of a latent trait to organize and evaluate the objec- 
tive measurement of that trait. A simple method for estimating item 
difficulties from observed item scores and person abilities from all 
possible person scores will be presented in terms of a short conven- 
ient FORTRAN! program compilable on most computers. Thus participants 
will be able to try the technique immediately ?.t their own institutions, 

Benjamin Wright 

University of Chicago 

4 hours 



Wednesday, February 28, 1973, 
Meeting Room 10, Jung Hotel 



2:15-6:15 
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TO: 



Past AERA Presession Directors 



FROM: 




SUBJECT: 



DATE: 



June 



1972 



In order to improve the research competencies of its 
members, AERA is sponsoring mini-courses during its 1973 
convention in New Orleans. These mini-courses will differ 
from presessions in at least three important ways: 

1. The mini-couises will be short — two to four hours 
in length. 

2. The mini-courses are intended to transmit specific 
skills in research, development, and evaluation 

to participants rather than to provide for a 
general discussion. 

3. The mini-courses will be held during the annual 
meeting at the convention site. 

Because you have already developed training materials 
for a much longer period of time, it may well be that 
some portion of that instruction would be self-contained 
and appropriate for such a mini-course. If you are 
interested in serving as an instructor, please prepare a 
short statement specifying what the participants will be 
able to do by virtue of attending the instructional session. 
The intended skills to be developed should be identified 
in as specific terms as possible. By July 15, 1973, m.ail 
this statement and a self-addressed envelope to me at 
Stone Hall, Cornell University, Ithaca, New York, 14850. 

Mini-course instructors will receive a modest 
honorariijm and/or travel expenses; participants will be 
expected to pay a small fee. 



JM:bl 



Now York State College of Agriculture and Life Sciences 

a Statutory College ol tho State University 
Cornell University 



Deparlmcnt of Education 

Stono Hall 

imaca. N. Y. 14850 



August 8, 1972 



Dr. Richard Anderson 
8 Lincoln Hall 
University of Illinois 
Urbana, Illinois 61801 

Dear Dxclc • 

This year the AERA Standing Committee on Research Training is ex- 
ploring ways to use the 1973 annual meeting (New Orleans - circa 
3/1) to provide research and research related skills to partici- 
pants. One (of three) such ways is to hold "conversation hours" 
in which prominent researchers like yourself spend up to Ih hours 
in informal discussion with young researchers who want to pick 
your brains. 

These conversation hours will differ from the graduate student 
seminars in two respects: (1) they will be open to all partici- 
pants, not just graduate students; and (2) they will focus on 
the specific research concerns of the participants rather than 
emphasize more general commentary. (The names of the roughly 
four conversation hour leaders will appear in ER, and conference 
goers will be encouraged to send you in advance ~2--page statements 
of problems they'd like you to discuss. Hopefully, your remarks 
will be addressed to some of these concerns.) 

Since this is our first attempt at something like this, we are 
most anxious to obtain outstanding individuals like yourself. I 
hope you will agree to participate. Please let me know as soon 
as possible if you are willing to undertake this virtually no*^ 
preparation "assignment". If your answe.r is positive, I'll need 
your address for 1/1/73-3/1/73. 



Cordially, 



JM;JBT 




(For the Committee) 



cc :_^Williara Russell 
W. James Popham 
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CONVERSATIONAL HOURS 



In an effort to explore ways in which the annual meeting can be 
used to increase the research competencies of its members, AERA is 
conducting conversation hours between prominent members of the 
educational research community and other researchers. 

The following are the discussion leaders, topics and location of 
the 4 conversational hours: 

Research on Human Learning and Memory: The Psychology of Instruction 



Discussion Leader: Richard Anderson 

University of Illinois 



Time: Wednesday, February 28, 1973, 2:15-3:45 

Location: Meeting Room 5, Jung Hotel 



Research pn Educational Measurement 



Discussion Leader: Robert L. Ebel 

Michigan State University 



Time: 
Location: 



Thursday, March 1, 1973, 10:35-12:05 
Meeting Room 5, Jung Hotel 



Research on the Conditions for Learning and Instruction 



Discussion Leader: Robert Gagne 



Center for Advanced Study in 

che Behavioral Sciences and 
Florida State University 



Time: 
Location: 



Tuesday, February 27, 1973, 12:25-1:55 
Meeting Room 4, Jung Hotel 



Research on Higher Ed' 



ucation 



Discussion Leader: Roger Heyns 



American Council for Education 



Time: 
Location: 



Wednesday, February 28, 1973, 2:15-3:45 
Mardi Gras H, Marriott Hotel 



Annual Meeting Notices 



Training 

ConvcrsatEonai Hours 

I'our sessions have been schcd-' 
ulcd during ihc Annual Mccving to 
facilitate conversations between 
prominent members of the educa- 
tional research community and 
• other researchers. Individuals who 
wish to attend one of these sessions 
are urged to write directly to the 
leader and indicate the research 
problem(s) which they would like 
addressed. The intended Advantage 
of this procedure is to insure that 
the majority of interests from the 
audience will receive a considered 
response froiii the discussion leader. 
Following arc the discussion leaders, 
topics they will consider and mail- 
ing addresses. 

• Richard Anderson, Lincoln 
Mall, University of Illinois, Urbana, 
UHnois 61801. (Research on Human 
Learning and Memory: The Psy- 
chology of Instruction.) 

• Robert L. Ebel, 449 Erick.son 
Hall, Michigan State University, 
East Lansing, Michigan 48823. 
(Research on Educational Measure- 
ment.) 

• Robert Gagnc, Center for Ad- 
vanced Study in the Behavioral 
Sciences, 202 Junipero Scira Blvd., 
Stanford, California 94305. (Re- 
search on the Conditions for Learn- 
ing and In^iruction.) 

• Roger lieyns, American Coun- 
cil for Education, One DuPont Cir- 
cle, Washington, D.C. 20036, (Re- 
search on Higher Education,) 

If you have a research problem 
in one of the areas represented 
above and if you*d like the man 
listed to discuss the problem with 
you, then send him a one or two- 
page letter indicating the concerns 
you*d like addressed. Mark the 
letter "AERA Conversation Hour." 

Exhibit of Self 
Instructional Materials 

Several types of self instructional 
materials will be available, free of 
charge* to registrants of the Annual 
Q ' wting. A permanent training ex- 



Programs 
for the 

1973 AERA Annual Meeting 

February 25 — March I 

New Orleans 
Have been mailed to alt 
AERA members 

BRING YOUR PROGRAM 
TO NEW ORLEANS 

Only a limited number of 
copies will be available in 
New Orleans. Advance regis- 
trants requiring an additional 
copy of the program will be 
charged a fee. 

Advance registration closes 
January 20. Registration and 
housing forms are in the 
program. 



hibit will be located in Mardi Gras 
D of the Marriott Hotel from 9-5 
during the four days of the meeting 
to enable individuals to enter Ihe 
exhibit and carefully examine and/ 
or study the training materials se- 
lected for display. Included among 
items displayed will be examina- 
tion copies of all AERA training 
tapes. 

Mini Research Training 
Courses 

The following 3 or 4 hour training 
sessions to transmit specific skills 
in research, development and evalu- 
ation have been scheduled during 
the Annual Meeting. Participants 
may register on site and will be 
assessed an $8 fee. 

Nonparamctric Analysis of Vari* 
ancc on Main Effects and Inter* 
action Based on Normal Scores (3 . 

hours) 

Instructor: Leonard A. Mara- 
scuilo. University of California at ' 
Berkeley 

Statistical tests, planned and post 
hoc methods of analysis based on 



the use of normal scores in place of 
original observations will be pre- 
sented for one, two, nested, and 
crossed ANOVA designs. 
Computers in Educational Research 
(4 hours) 

Insiruciors: Ronald G. Ragsdale 
and Sorcl Reisman.. Ontario Insti- 
tute for Studies in Education 

This is a computer literacy course 
that will enable participants to 
generalize about and discriminate 
between, computer systems, options 
and languages and make intelligent 
decisions about their uses. 
Plannin;; Formative Evaluations for 
Instructional Products (3 hours) 

Instructors: Richard A. McCann, 
Eva L. Baker, Evan Keislar, and 
Merlin C. Wittrock, University of 
California at Los Angeles. 

The trnining session will (I) in- 
troduce the participant to the major 
considerations involved in planning 
a formative evaluation — statement 
of purpose, selection of measures, 
sample, design, method of analysis, 
criteria for decisions to revise, ad- 
ministrative plan, and (2) provide 
the participant with experience in 
reviewing and criticizing summaries 
of formative evaluation plans. 

Sample-Free Item Calibration and 
Test-Free Person Measurement (4 
hours) 

Instructor: Benjamin Wright, 
University of Chicago 

A simple method for organizing 
and evaluating the objective meas- 
urement of a latent trait presented 
in the language of a short conven- 
ient FORTRAN program. 



Stanford No Host Dinner 

A no-host cocktail hour and din- 
ner for faculty, students, alumni 
and friends of the School of Educa- 
tion, Stanford University, will be 
held Monday, February 26, at the 
Royal Orleans Hotel in New Or- 
leans. Cocktails (no-host) will be 
served from 7:30; dinner at nine. 
For reservations write by February 
I, to Ms. Doris Fevrier, Schooi of 
Education, Stanford University, 
Stanford* Calif. 6430S. 



NEW YORK STATE COLLEGE OF AGRICULTURE 
A STATUTORY COLLEGE OF THE STATE UNIVERSITY 
CORNELL UNIVERSITY 




DEPARTMENT OF EDUCATION 
STONC HALL 



Dear Col league: 

In an effort to explore ways in v;hich the Annual Meeting 
can be used to improve the research, development and evaluation 
skills of its members, the American Educational Research Association 
will sponsor a self-study center at the 1973 New Orleans meeting. 
The center will consist of film materials, audio tape cassettes, 
and workbook supplies which provide self-instructional training 
in research, development and evaluation skills. AERA staff members 
will moniter the materials. A one-time only fee of $K00 v/i 1 1 be 
charged to users; publishers of commercially available materials 
will be charged $25.00 if any of their products are displayed. 
The room wi 1 1 be open for several hours each day of the convention. 
Except for filmed materials v/hich will be shown at scheduled times, 
all selected products will be available for self-study whenever 
the center is open. 

If you have authored materials v/hich you would like used 
at this convention and which you feel are self-instructional and 
provide for R, D or E ski 1 1 training , please send me a specimen 
set of the instructional products in time to reach me by September II, 
1972 and note v/hether multiple copies of non-film products can be 
provided for the New Orleans convention^ (These materials will be 
returned shortly after they are received, at which time you will be 
notified if they are judged appropriate for the self-study center.) 

I welcome correspondence or telephone calls if you wish to 
discuss this project further. If you know of other self-Instructional, 
R, D e E skill training materials of high quality which you think 
appropriate for our needs, won't you be, good enough to provide me 
with an appropriate name and address? 

Cordial ly , 

'Jason Mi 11 man, Member 
AERA Standing Committee on 
Research Training 
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EXHIBIT OF SELF- INSTRUCTIONAL MATERIALS 



MARDI GRAS D, M-IRRIOTT HOTEL 
9-5 Monday through Thursday 



THE FOLLOWING SELF- INSTRUCTIONAL RESEARCH TRAINING MATERIALS 
WILL BE AVAILABLE FOR INDIVIDUAL STUDY AND /OR EXAMINATION: 
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Title: 



Perspectives on Recent Research AERA Cassette Tape 
Recordings 



Authors: 



R. Gagne, D. Meyer, J. Piatt, A, A. Liimsdaine, J. Millman, 
M. Scriven, R. Glaser, F.N. Kerlinger, £• Rothkopf, 
R. Travers, N. Flanders, R. Jones, H. Levin, L. Marascuilo, 
R. Anderson, 



Description: 



Each tape by an eminent scholar in the field was commissioned 
by AERA with a specific objective to give the listener at 
least one important technical skill relating to educational 
research. Although primarily intended as an update device 
for the educational researcher who has completed his formal 
training, many professors will find the tapes ideal for 
their graduate classes. 



Title: 

Authors: 

Description: 



Applied Decision Making 

Joseph Ward Jr. , Air Force Human Resources Lab; and AJERA. 

A prototype of this instructional package is available for 
review. It is designed to train the decision maker in 
knowing when data collection can aid in the decision, making 
process, and 2) knowing what information to communicate to 
the resource person so that an appropriate design is utilized 
and the data is analyzed meaningfully. Ample use is made of 
examples and simulations. 



Title: Evaluation Workshop I 

Author: Stephen P. Klein, UCLA. 

Description: Evaluation Workshop I is a structured 2 day workshop experience 
which, through the process of instruction, practice, feedback, 
and discussion, provides participants with an orientation to 
evaluation that is invaluable for informed educational decision 
making at all levels. The workshop materials are on display. 
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The Educational Information Consultant: Skills in Disseminating 
Educational Information 

Bela Banathy, et al.. Far West Laboratory for Educational 
Research and Development 

The EIC Program is designed to train individuals to: analyze 
and define an educator-client's information problem; plan and 
execute a search strategy; screen, sort, and package the in- 
formation; communicate it to the client; and evaluate how effec-^ 
tive the serv^ice has been. It is available in a Course form, a 
ten-day Institute form, and a self-instructional Learning Team 
form. Its modular format permits adaptations to specific 
audiences and contexts. 



Title: Determining Instructional Purposes Training Package 

Author: Educational Meangement Program, Far West Laboratory for Educa- 

tional Research and Development 

Description: Determining Instructional Purposes is an instructional manage- 
ment training package designed to help school decision makers 
establish and validate the goals and objectives of their in- 
structional programs. It consists of: 

1. An orientation unit providing an overview of the 
processes of determining instructional purposes and 
describing the three units that deal with these pro- 
cesses. 

2. Three training units (Setting Goals, Analyzing 
Problems, and Deriving Objectives), each providing 
10-15 hours training in the basic knowledge and skills 
involved in determining instructional purposes. 

3. A Coordinator handbook thut presents procedural 
guides and suggestions for setcing up and conducting 

a training program using one or more of the basic units. 



Title: 
Authors : 
Description: 



Title: 



An Interviewing Training Module 



Author: Emil Haller, Cornell University 

Description: The module is designed to teach students the technique 

of research interviewing. The module requires a maximum 
of 3 classroom hours. It consists of 3 films, manuals, 
role-playing simulations and practice schedules. 



Title: Appraising Educational Research: A Case Study Approach 

Authors: D. Bob Gowin and Jason Millman; Cornell University 

Description: This self instructional material asks the learner, in a semi- 
structured format, to appraise critically each of nine re- 
search articles. Also provided are model critiques developed 
with the assistance of over 800 students from 27 colleges 
and universities. 



Title: Research-Based Techniques for Instructional Design 

Authors: Eva Baker, UCLA; and Edys Quellmalz , SWRL 

Description: The materials consist of 12 hours of instruction, text 

discrimination, and practice simulations designed to teach 

instructional materials developers to write first draft 
sequences incorporating given techniques. 



APPENDIX D 



MODEL D APPENDIX 



Appendices for Model D (see report of model on pages 19-42) are 
available upon request under seperate cover. 



Instructional booklets developed under Model D are included under 
seperate cover or available upon request* 



Instructional Cassette Tape recordings developed under Model are 
included under seperate cover or available upon request. 
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REPORT OF AERA SURVEY ON EVALUATION OF 
AERA EXPERIMENTAL TRAINING SESSIONS 
(OCTOBER, 1973) 



Robert M. Stonehill 



Laboratory of Educational Research 
University of Colorado 



Description of the sample 

Copies of the questionnaire and cover letter signed by W. James Popham 
were sent to 600 AERA members according to a randomly selected Mailing list 
supplied by William Russell at the AERA office in Washington, D,C, A ssunple 
copy of the questionnaire with indicated frequencies of resi>on3e for each item 
and item means where appropriate is included at the end of this report. 

The questionnaires were mailed from Boulder, Colorado on June 22, 1973« 
By August 6, 350 responses had been received. Of these, 11 were dropped from 
the study because they had been left blank or they consisted of empty envelopes. 

Although anonymity was maintained on the questionnaire itself, a system 
of nvunbering the return envelopes enabled the researcher to keep track of 
who had responded to the survey. The questionnaires were removed from the 
envelopes as soon as the response was checked off. 

On August 15 a- follow-up mailing was sent to the 250 people who had not 
responded. Of these 250, 88 more responses were received, resulting in a total 
return of ^38 (of which ^2? were included in the data analysis), or 7% of 
the sample of 6OO. 

The survey included the entire United States, with the subjects of the 
survey selected proportionately by state- 

Purpose of the survey 

The purpose of this survey is to determine whether or not the present 
AERA training institutes (traveling institutes, short courses such as the 
pre-sesslon and post-session offerings at the AERA annual convention in 
New Orleans, training materials exhibits and conversation hours) are addres- 
sing themselves to the AERA membership in the following wayst 

1. Do they address themselves to members of all AERA divisions? 

2. Are the topics offered representative of those with the highest level 
of interest among AERA members? 

3. Are the various types of training institute cost-effective from the 
point of view of the potential participant? 

^. Are the times these institutes are being offered the most efficacious 

for those likely to attend? 
5. Are those who attend various types of training insttiutes "typical** 

AERA members? 
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DATE OP RETURNS 



TaMe 1, Frequency of questionnaire returns vs» elapsed time from malllng> 



Description of the respondents 

There were no significant differences In response rates across different 
geographic regions of the country. Of the 438 responses, 11 were not Included 
In the data analysis. Among the 427 remaining returnees, the dominant primary 
employment was In a college or university (266), followed by those employed 
In an elementary or secondary school system (59) • The professional activity 
most often participated In was research or evaluation (335 people, with a 
mean of 3256 of their professional time spent in that fvnction) followed by 
instruction (274 people, with a mean of 4556 of their professional time spent 
in instruction). 

The predominant AERA division in which respondents held membership was 
Learning and Instruction (i83), followed closely by Measurement and Research 
Methodology (176). 

There seemed to be no significant differences between those respondents 
answering the first mailing and those answering the follow-up mailing. Many 
of those who did not respond the first time indicated on their follow-up 
return that they were away for the sumraert 
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Traveling Institutes 

315 people were aware of the series of AERA sponsored traveling institutes, 
266 of whom first heard of them through advertisements in Educational Researcher 
journal. Only 21 people first became aware of the traveling institutes through 
one page fliers distributed by AERA. 

Of '%15 poople who were a>?are of the traveling institutes, only 16 actually 
attended any. The main reason cited for returnees not attending any traveling 
institutes was "I could not take off from my ,1ob long enough for the Institute 
^142)" and "the institute in >diich I was interested was held too far from my 
home (103).'* 90 people felt that the Institute cost more than they felt it would 
be worth. The least important reason for people not attending was that 
"the Institute directors were unknown or unimpressive to me," with 9 people 
responding to that category. 

Contingency tables were used to examine further the characteristics of 
those who were aware of the series of traveling institutes and of those who 
actxaally attended traveling institutes (see Contingency Tables 1,2 and Al-), 

Across all types of employing institution, the percentage of those who 
were aware of the traveling institutes was not significantly different 
(chi-square « I3.6 with 5 degrees of freedom). The mean percentage of those 
aware of the traveling institutes was 73»8^» The State Department of Education 
category yielded the highest proportion of awareness (92.9^, or 13 out of Ik) 
of the existence of the traveling institutes, followed by College or University 
(75.6?gf or 201 out of 266). 

The amount of people who attended any of the traveling institutes was 
extremely small as a i>ercentage of the population sampled (16 of ^27, or 
3.72) • They did not belong to any clear cut type of employing institution, 
though no one from a Regional Laboratory or R & D Center or from a State 
Department of Education sampled had attended a traveling institute • A person 
attending a traveling institute seemed more likely to attend the annual AERA 
convention in New Orleans than a person who did not attend a traveling institute. 

Convention related activities (pre'^session short courses, post^session short 
courser>, mini "-courses, conversation hours and training exhibitsl 

The number of respondents who had attended the AERA convention in New 

Orleans was 13?, which was 30.956 of the sample. Of this number, 21 attended 

one or more training exhibits, 10 attended at least one mini -course, 26 

attended one or more conversation hours, and only 8 attended either a pre- 

sesrlon or a post-session short course. 

ERLC 



The primary reasons K^ven for not attending any of the New Orleans training 
serir;ionr=: were: inability to be in New Orleans at the time of the training 
sesfiions (I7l)f inability to be in New Orleans early enough to attend a 
pre-session short course (68), and inability to remain in New OrleanvS long 
enough to take a post-session short course (62), 50 people found themselves 
too busy during the convention to attend a mini -course, conversation hour or 
training exhibit, 

A contingency table analysis was performed comparing type of employing 
institution with awareness of pre-session or post-session short courses 
(see Contingency Table 3), 35^ people (82,95?!) were aware of the existence 
of the short courses. People employed in Regional Educational Laboratories 
or B.&D Centers tended to have the smallest proportion (58.^) of those 
aware of the short courses. I3 out of Ik employees of a State Department 
of Education (92,99^) and 232 out of 266 employees of a College or University 
(87.2^) were aware of the short courses* Non-awareness of the existence 
of the pre-session and post-session short courses did not seem to be a 
significant reason for people not attending them. 

There was no contingency table analysis performed on those who attended 
pre-session or post-session short courses, since only 8 people attended any. 

Recommended fee structuTC and time of year 

In response to the question "During lihioh times of the year are you most 
available to attend an AERA training session?", 218 people favored the summer- 
time and 144 favored winter. Only 95 people felt the springtime to be most 
efficacious, and fall was rated least desirable, with 89 people responding 
in that catef2;ory. 

People tended to feel that a fee structure of approximately $25/day 
would be the most appropriate for an AERA training session, 23I people 
responded to $20 - $30 per day, with the frequency of response dropping 
drastically after that: 75 people checked $30 - $^0, and only 1 person 
felt a fee of between $60 and $70 to be reasonable. 

Recommended topics 

The only open-ended question of the survey vas a request for suggestions 
of topics that would be appropriate for an AERA-sponsored training session. 
Following is a list of the most frequently encountered suggestions! 

ERLC 



5 



Problems In test validity and reliability 

Simulated research and evaluation problems in public schools 
Problems of dealing with sex biases 

Researching the difficult -to -measure variables in education 

Specifying outcomes for humanistic programs 

Program evaluation 

Behavior analysis in education 

Accountability systems for teacher training programs 
Measuring learning outcomes 
Kanagement of Instruction 
English as a second language 

Advanced topics in statistics and research design 
Program planning 
Test construction 

Evaluation in affective areas (art, literature, etc.) 
Multivariate statistics 
Time Series Analysis 

Use of computers to solve research problems 

Many people suggested topics which were already being offered in 
AERA training institutes, such as non-i)arametrlc statistics. Other people 
expressed satisfaxstion with the range of topics offered, citing other reasons 
for their inability to attend. 



Additional analyses 

A random half of the sample were sent return envelopes with stamps 
affixed, while the other half were sent return envelopes with no stamp. 
All had return addresses, as well as a number to identify the respondent 
for purposes of the follow-up mailing. 18? returned pre-stamped envelopes 
while 163 returned envelopes which had not been stamped by the researcher. 
A chi-square analysis with 2 degrees of freedom failed to detect a significant 
difference between these two frequencies, and no return envelopes had stamps 
put on them for the follow-up mailing on August 13* This seemed in accordance 
with earlier studies (Brzezlnskl and Worthen) , but cannot be generalized to 
all classes of questionnaire survey respondents. Most people on the faculties 
of colleges, universities, State Departments of Education and the Federal 
Government have mailing privileges. 

At least two people wrote. rather acerbic comments saying that AERA 
should hiive the decency to affix stamps if they expected people to do them 
the favor of filling out their questionnaires, but there is no way of knowing 
if anyone did not return their questionnaire for this reason. 
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nONTTNGENGY TABLE 1: EMPLOYING INSTITUTION VB. A'^ARENSSS OF TRAVELING INSTITUTES 



AWARENESS OF 
TRAVELING 
INSTITUTES 



EMPLOYING 
INSTITUTION 


NOT 
AWARE 


AWARE 




COLLEGE OR 
UNIVERSITY 


65 


201 
(75.6^) 


266 
(62.36) 


ELEMENTARY OR 
SECONDARY 
SCHOOL SYSTEM 


17 

■ 

(28.65^) 


42 
(71.2?6) 


59 
(13.8P6) 


REGIONAL 

ITTlTTr'ATTAWAT 
aUu UA 1 X v/nALr 

LABORATORY OR 
R&D CENTER 1 


6 


6 

(50^) 


12 
(2.8^) 


STATE DEPT. ' 
CF ' 
EDUCATION 


(7.156) 


(92.9^) 


14 
(3.3^) 


FEDERAL 
GOVERNMENT 


(44.4^) 


5 

(55.6^) 


9 

(2.1S6) 


OTHER 


19 
(28.:^) 


•+(3 

(71.7%) 


(15.756) 




112 
(26.2^) 


315 

(73.856) 


427 
(100^) 
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CONTINGENCY TABLE 2t EMPLOYING INSTITUTION VS. ATTENDANCE OF TRAVELING INSTITUTES 

ATTENDANCE OF 
TRAVELING 
INSTITUTE 





NO 


YES 




COLLEGE OR 
UNIVERSITY 


256 
(96.2%) 


10 
(3.8%) 


266 
(62.3%) 


ELEMENTARY OR 
SECONDARY 
SCHOOL 
SYSTEM 


58 

(98.3%) 


1 

(1.7%) 


59 
(13.^) 


REGIONAL 
EDUCATIONAL 
LABORATORY OR 
R&D CENTER 


12 


0 


12 
(2.8%) 


STATE DEPT. 
OF 

EDUCATION 


Ik 


0 


Ik 
(3.^) 


FEDERAL 
GOVERNMENT 


8 


1 


9 


OTHER 


58 
(93.5%) 


(6.5%) 


62 
(1^.5%) 




(96.3%) 


16 

(3.7%) 


kZ7 
(100%) 



CONTINGENCY TABLE 3t EMPLOYING INSTITUTION VS. AWARENESS OF PRS-SES3I0N AND 

POST -SESSION SHORT COURSES 

AWARENESS OF 
SHORT 
COURSES 





NOT 








AWARE 


AWARE 




COLLEGE OR 
UNIVERSITY 


1 

lit 

(12.^) 


232 

(87.2^) 


COO 

(62.3^) 


ELEMENTARY OR 
SECONDARY 


13 




59 


SCHOOL 
SYSTEM 


(2256) 


(78^) 


i (13. a^) 

i 
1 

\ 


REGIONAL 








EDu UAT 1 Ori Ail 


5 


7 


12 


LABORATORY OR 
R&D CENTER 




(58.3^) 


(2.a?g) 


STATE DEPT. 


1 


13 


14 


OF 

EDUCATION 


(7.W) 


(92.9^) 


(3.3?6) 


FEDERAL 


3 


6 


9 


GOVERNMENT 


(33.3^) 


{67%) 


(2.1^) 




17 


50 


67 


OTHER 




(7^,6%) 


(15.7^) 




?3 
(17.1^) 


35^ 
(82.9^) 


427 
(100^) 
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CONTINGENCY TABLbl 4» ATTENDANGl!: OF ANNUAL A£HA CONVENTION (NSW ORLSANS) V3 . 

ATTENDANCE OF TRAVELING INSTITUTES 



ATTENDANCE OF AERA 



ATTENDANCE 


CONV 


SNTION 


OF 

TRAVELING 
INSTITUTE 


NO 


YES 


NO 


286 


125 


(69.6%) 




YES 


9 


7 


(56. 35^) 


(^3.856) 



295 
(69=15^) 



132 
(30.9%) 



411 
(96.3%) 

16 
(3.7%) 

427 
(100%) 
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ameRican eoucaTionaL 
ReseaRCH associaiion 



Qt^r Colleague: 



During 1972 and 1973, AERA has conducted several experimental methods 
for providing training opportunities in research and research-related areas. 
These methods include traveling institutes^ experimental presessions held in 
conjunction with the annual meeting, and new training opportunities at the 
annual meeting (e.q,, mind-coursest conversation hours, and exhibits of 
training materials). 

The AERA Research Training Committee is currently evaluating these experi- 
mental training methods to assist in the planning of subsequent training sessions 
to be sponsored by AERA. Your name was drawn as one of a small random sample of 
AERA members from whom we are eliciting information for that evaluation. As the 
committee member responsible for this evaluation, I would like to ask you to 
help us by filling out and returning the enclosed questionnaire, whether or not 
you have participated personally in any of the experimental training sessTohsT" 

The questionnaire is brief and should take only a few moments to complete. 
I know your time is limited, but your response is important to our conwittee 
in completing this evaluation and I hope you will take the time necessary to 
help in this endeavor. 

The number included on the questionnaire is only for purposes of checking 
off those who have responded. Your reply will be held in strict confidence and, 
to Insure that, all numbers will be removed from incoming questionnaires as 
soon as they are checked off. The analysis and tabulation of responses will 
be done for us by the Laboratory of Educational Research at the University of 
Colorado, and all completed questionnaires should be returned directly to them. 

Please return the questionnaire in the enclosed addressed envelope. The 
committee and the Association will be grateful for your response. 



Sincerely yours. 




W. James Popham, Chairman 

AERA Research Training Committee 



Enclosures 
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frequency of response 
mean of that variable 



EVALUATION OF AERA EXPERIMENTAL TRAINING SESSIONS 



Directions : Please read each question carefully and please be completely 
candid 1n your responses. Your name will never be associated with the responses 
you provide. 



1. Your primary employment is presently 
In a (check ONE): 



(266) col lege or universi ty 
(59j^elementary or secondary school 
system 



What portion of your professional 
time do you spend in each of the 
fol lowing activities: 



(274)lnstruction ^5 % A^m^g^stration 37 % 



(12) regional educational laboratory (l68)Development or 
— ^ . r.^r. Dissemination % 



or R&D center 
(1^) state department of education 
(9) Federal government 
(6) 'Other (please specify) 



(335)Research or 
Evaluation 



32 % 



Student (formally 
eijg^^led) Jj6% 

Other ( ) 



(59) 
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Please check the AERA Division(s) in which you presently hold membership 
(check an^ that apply): 

(78)\. 
(r3^J)B. 

(fT^D. 

(5r F. 

(53)" G. 

(sor H. 



(Administration) 

(Curriculum and Objectives) 

(Learning and Instruction) 

(Measurement and Research Methodology) 

(Counseling and Human Development) 

(History and Historiography) 

(Social Context of Education) 

(School Evaluation and Program Development) 

Student member 



Were you aware of the AERA-sponsored series of Traveling Institutes (courses 
in research, evaluation, or related topics, repeated in various parts ^pf the 
country)? 



(315)YES 



(112)nO 



If YES, where did you first hear of the Traveling Institutes? 
(266£clucational Researcher journal 

single-page "flyer" 
(27) other (please specify) 



(check ONE) 



b. If NO, skip to question 6, 



Did you attend any of the Traveling Institutes? 

(li)_YES (^11) NO 

a. If YES, which one(s) did you attend? 



(DirectorsT 



(Location) 



(Directors) 



(Location) 



-2- 

b. If NO, please check below your reason(s) for not attending (you may 
checF up to THREE): 

(QOj The topics offered did not meet my needs or were uninteresting 

to me. 

(90) The Institute cost more than it seemed to be worth. 
(^^J I could not take off from my job long enough for the Institute. 
(1637 The Institute in which I was interested was held too far from my 
home . 

(65l_ I had a previous commitment which prohibited my attending the 
Institute in which I was interested. 

(9) The Institute directors were unknown or unimpressive to me. 

(^571 Other (please specify) 



6. Did you attend the annual AERA convention in New Orleans? 

( 132) YES (295) NO 

7. Whether or not you attended the annual convention, were you aware of the 
pre-session and/or post-session short courses offered by AERA in New 
Orleans? 

( 35^) YES (73) NO 

8. Did you attend any pre- or post-session short courses? (8) YES(^19) NO 



a. If YES, which one(s) did you attend? 



9. If you were in New Orleans at the time of the annual meeting, did you attend 
any training exhibits, mini-courses or conversation hours? 

Training exhibits Mini -courses Conversation Hours 

Y ES (406) ^NO (10) YES NO (^1?) (26) YES (^01) NO 



10. If you did not attend any training courses in New Orleans, please indicate 
your reasons for not attending (please indicate all reasons that apply): 

(171) I could not be in New Orleans at the time of the training sessions. 
(6S]2I I could not be in New Orleans early enough to take a pre-session short 
course. ' 

(62) I could not remain in New Orleans long enough to take a post-session 
short course. 

(3 3) I did not find that any of the offered topics was relevant to my needs. 
(5qj3 During the convention, I was too busy to attend a mini -course, conver- 
sation hour, or training exhibit. 
(4) The people offering the courses were unknown or unimpressive to me. 
(527^ Other (please specify) 



VI. During which times of the year are you most available to attend an AERA 
training session? 

(144) WINTER (95) SPRING (218) SUMMER (89) FALL 
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12. What do you think would be the most appropriate fee structure per day 
for an AERA training session? 

(231) $20-30 (75) $30-40 (^3) ^$40-50 W ^$50-60 (1) $ 60-70 

X = $25/day 

13. If you could choose, what topic(s) would you suggest for an AERA-sponsored 
training session? 



THANK YOU FOR YOUR COOPERATION 
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